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EXECUTIVE  SUMMARY 


OBJECTIVE 

The  User-Centered  Design  and  Engineering  (UCD&E)  team  constantly  is  improving  the  project 
management  tool  called  the  portal.  The  portal  is  a  web  interface  SSC  Pacific  software  solution  for 
project  managers.  Based  on  web  navigation  schemes,  it  follows  time-tested  human  factors  of  web 
design.  Successful  software  must  be  easy  to  use,  ergonomic,  and  simple  in  its  design.  This  report 
details  research  the  UCD&E  team  conducted  to  improve  those  three  areas  in  the  portal  software 
interface. 

METHODS 

The  report  follows  research  conducted  in  many  areas.  Wireframe  usability  tests  conducted  in  April 
2015  helped  isolate  user  design  issues  that  once  resolved  made  using  the  portal  software  much  more 
intuitive.  Usability  tests  conducted  in  May  2015  focused  on  improving  navigation,  interface  design 
and  user  task  performance.  Part  of  the  study  included  an  A/B  study.  The  UCD&E  team  used  an 
eight-question  survey  to  compare  four  competing  design  options  to  understand  which  of  four 
methods  of  secondary  portal  navigation  users  deemed  “easier  to  use.”  The  result  of  the  A/B  study 
showed  two  choices  testing  superior  to  the  rest.  Those  solutions  are  detailed  in  this  report.  Testing 
included  rapid  user  testing.  Rapid  user  testing,  allowed  suggested  user  changes  to  be  made  to  the  user 
interface  to  be  made  as  soon  as  a  problem  is  identified.  This  allowed  the  solution  to  be  validated 
quickly  whether  the  solution  was  successful  or  needed  further  work.  Research  captured  a  majority  of 
user  experience  issues. 

CONCLUSIONS  AND  RECOMMENDATIONS 

Based  on  research  findings,  the  UCD&E  team  found  flaws  in  the  software  and  suggested 
improvements  for  the  software  development  team.  Issues  found  in  the  portal  software  study  will  be 
used  to  improve  new  versions  of  the  portal  software. 

Overall,  usability  test  results  revealed  that  the  interface  design  and  the  planned  interaction  models 
were  highly  successful.  Users  who  attempted  to  create  a  project  using  methods  provided  in  the 
project  portal  interface  were  successful.  Every  user  completed  the  task,  with  minimal  delays  or 
confusion,  and  expressed  confidence  in  repeating  the  task  in  the  future.  There  were  no 
user-experience-related  mistakes.  Clear  portal  definitions  and  instructions  led  to  successful  user 
performance.  The  Input/Output  screen  transitions  and  graphic  user  interface  wizards  performed 
according  to  users’  expectations.  Users  could  navigate  successfully  through  portal  software  solution 
paths  and  could  complete  tasks  in  every  instance.  Users  appreciated  the  ease  and  forethought 
designed  into  the  portal  as  a  whole,  and  the  straightforward  nature  of  the  graphic  user  interfaces. 
Users  were  delighted  with  the  look  and  feel  and  felt  optimistic  that  they  could  use  the  interface  to 
perform  their  project  management  tasks.  Users  successfully  completed  the  tasks  tested  in  any  given 
round. 
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1.  PRODUCT  AND  DESIGN  GOAL 

The  User-Centered  Design  and  Engineering  (UCD&E)  team  joined  the  Process  Architecture 
Improvement  project  to  employ  user  research  and  iterative  user-focused  testing  to  help  design  a 
single  space  for  Space  and  Naval  Warfare  System  Center  (SSC  Pacific)  project  managers  to  access 
the  project  management  content  necessary  to  meet  Center  requirements.  Throughout  this  process, 
user  feedback  obtained  presented  data-based  justification  for  a  hands-on,  customized  project 
management  experience  based  on  characteristics  unique  to  each  project  and  project  manager. 

The  portal  is  a  project  management  tool  based  on  web  navigation  schemes,  and  followed  time- 
tested  human  factors  heuristics  for  web  design.  An  example  of  the  portal  software  interface  is 
provided  in  Appendix  E,  where  several  sample  screens  give  a  preview  of  the  portal’s  appearance  and 
how  it  is  organized  and  used. 

Portal  test  results  detailed  in  this  report  focus  on  many  areas  specific  to  improving  the  portal  and 
incorporating  user  feedback.  The  UCD&E  team  conducted  multiple  rounds  of  user-centered  research 
that  incorporates  structural  and  content  requirements  provided  by  the  product  team  for  inclusion  in 
the  portal.  This  research  assessed  the  product’s  navigation  elements,  its  consistent  use  of  language, 
how  well  it  supports  shared  understanding  of  the  portal’s  underlying  concepts,  and  whether  it  would 
enable  users  to  meet  the  administrative  requirements  with  which  project  managers  must  comply. 

Subsequent  tests  focused  on  the  tasks  associated  with  creating,  joining,  and  managing  project 
resources  through  the  project  portal,  using  task-based  analysis  and  think-aloud  protocols  to  examine 
the  facility  with  which  users  performed  various  of  basic  tasks  required  in  project  management  work. 

The  UCD&E  team  conducted  A/B  comparison  tests  designed  to  examine  users’  preferences  and 
ability  to  properly  differentiate  between  various  methods  of  displaying  and  navigating  certain  design 
elements  within  the  portal.  An  overview  of  A/B  studies  is  provided  in  Section  4.  Detailed  results  of 
the  A/B  comparison  tests  are  provided  in  Appendix  C. 

Elements  of  navigation,  language  usage,  and  inter-portal  communication  were  tested  and  analyzed 
to  assess  successes  and  failures  of  several  iterations  of  portal  design.  Test  protocols  were  primarily 
designed  to  elicit  feedback  on  how  well  various  incremental  prototypes  of  the  project  portal  met 
participants’  mental  models  and  usage  expectations  of  how  the  software  should  work.  Additional 
goals  included  evaluating  the  navigational  elements,  wording  of  language -based  elements,  goal- 
based  content,  and  overall  usefulness  of  the  interface. 

Unfortunately,  not  all  of  the  feedback  gained  could  be  implemented  into  the  current  portal  design. 
To  the  extent  that  users  focused  on  tools  or  capabilities  not  in  the  current  scope  of  the  portal  design, 
the  UCD&E  team  documented  these  considerations  for  future  use  to  add  product  value  and  facilitate 
user  adoption  of  the  portal  for  future  portal  software  versions.  The  UCD&E  team  then  produced  and 
documented  iterative  design  recommendations  to  communicate  research  findings  to  the  product  and 
development  teams.  These  recommendations  may  not  be  fully  or  consistently  incorporated  into  the 
end  product,  due  to  decisions  and  constraints  beyond  the  control  and  influence  of  the  UCD&E  team. 
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This  inconsistency  may  prevent  the  portal  from  realizing  its  fully  designed  potential.  Research 
suggests  that  if  project  managers’  expectations  for  project  management  and  process  support  are  not 
fully  met,  they  may  lose  confidence  in  the  overall  product  suite  and  the  adoption  rate  would 
subsequently  be  low. 
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2.  WHY  UCD&E? 


SSC  Pacific’s  UCD&E  team  develops  solutions  for  a  variety  of  customer  needs,  from  developing 
in-cab  geo-visualization  systems  for  the  United  States  Marine  Corps,  to  designing  the  physical  layout 
of  an  emergency  operations  center,  to  prototyping  mission  planning  applications  for  the  submarine 
and  surface  communities,  and  more.  The  team’s  methodology  supports  product  design,  process 
improvement,  technical  writing,  and  instructional  design. 

At  the  most  basic  level,  UCD&E  is  about  the  user.  Through  user-focused  testing,  interviews, 
surveys,  knowledge  elicitation,  cognitive  walk-throughs,  task-based  analysis,  think-aloud  protocols, 
card  storming,  comparison  testing,  knowledge  mapping,  and  other  techniques,  UCD&E  researchers 
can  identify,  isolate,  and  focus  on  the  elements  most  basic  to  the  needs  and  desires  of  the  user 
population,  and  join  them  to  the  mission  goals  of  the  stakeholders.  By  designing  the  capabilities  of 
the  system  around  these  core  principles  and  metrics,  stakeholders  successfully  appeal  to  the  user 
base,  designing  a  functional,  desirable,  valuable  product  with  high  satisfaction,  low  levels  of  rework 
or  user  support,  and  high  adoption  rates. 

User-centered  design  is  an  approach  that  focuses  early  on  user  needs  and  subsequently  leverages 
sound  human  engineering  and  scientific  methodologies,  which  include  a  significant  amount  of 
analysis,  to  provide  early  definitions  of  user-focused  requirements,  rather  than  function-focused 
requirements.  By  employing  user-centered  design  (UCD)  methods,  we  enhance  measures  of 
effectiveness  and  measures  of  performance  and  improve  situational  awareness. 

The  UCD  method  involves  direct  knowledge  elicitation  sessions  with  the  users,  and  is  key  to 
developing  task-based  designs.  Through  this  method  the  team  develops  cognitive  models,  use  cases, 
and  prototypes  that  directly  support  rapid  development  and  other  agile  processes.  At  the  highest 
level,  UCD  products  focus  and  guide  integration  efforts  to  support  systems  engineering  towards 
achieving  total  system  performance,  and  at  the  lowest  level  directly  influence  the  mission 
effectiveness  of  each  person  using  tools  based  on  UCD  products  and  processes. 

The  UCD&E  team  exists  to  ensure  that  products  produced  throughout  the  Center  and  elsewhere 
within  the  Department  of  Defense  support  the  goals,  needs,  and  wants  of  end  users,  and  balance  those 
goals,  needs,  and  wants  with  those  of  the  stakeholders.  Through  proven  methodologies  used  by 
leading  companies  known  for  providing  exceptional  user  experiences,  the  UCD&E  team  develops 
requirements,  prototypes,  and  deployable  end-products  that  significantly  improve  end-user 
effectiveness,  increase  situational  awareness,  and  reduce  total  cost  of  ownership. 
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3.  DEMOGRAPHICS 


Throughout  these  studies,  the  UCD&E  team  maintained  strict  methods  and  protocols  for 
advertising,  selecting,  scheduling,  processing,  testing,  and  reporting  all  interactions  with  participants 
across  the  Center.  Thirty-two  SSC  Pacific  employees  (civilian  and  enlisted)  participated  through 
randomized  targeting  of  a  diverse  population  through  email  advertisements.  Participants  then  self- 
selected  and  scheduled  available  times  with  the  UCD&E  team  for  testing.  The  participants  were 
distributed  as  follows: 

•  Gender:  Both  male  and  female 

•  Experience:  Less  than  a  year,  up  to  16  plus  years 

•  Number  of  Concurrent  projects:  From  1-1 1  plus,  with  most  working  on  2-4  projects  at  a  time 

Figure  1  details  the  selection  of  participants  across  the  Center.  Selection  of  participants  was 
pseudo-randomly  distributed,  varying  across  experience  levels  and  competencies.  This  mix  of 
participants  gives  the  UCD&E  team  confidence  in  its  findings  and  that  results  found  are  sufficiently 
substantiated. 


Figure  1.  Participant  demographics. 
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4.  TESTING  RESULTS 


4.1  INITIAL  RESEARCH 

Initial  user  research  into  the  project  management  portal  indicated  that  the  User-Centered  Design 
and  Engineering  team’s  first  navigational  layout  and  workspace  iteration  was  moderately  successful. 
Users  largely  appreciated  the  goal  of  the  portal  and  the  design  of  its  layout.  Throughout  the  portal, 
however,  there  remained  a  fundamental  need  for  improved  clarity  across  the  interface  to  address 
shortfalls  with  the  overall  content,  task  completion,  and  value  proposition  for  the  user.  Subsequent 
design  focus  was  directed  at  improving  these  interactions  in  the  context  of  becoming  the  single 
interactive  source  for  all  Center  project  management  processes. 

Specifically,  the  areas  identified  for  improvement  after  the  first  round  of  testing  included  the 
following: 

•  Reassess  incorrect  assumptions  about  the  Center  project  managers 

•  Improve  language  and  navigational  clarity  regarding  content,  user  goals,  and  value 

•  Re-center  focus  on  product  management  processes  instead  of  day-to-day  project  management 
tools 

•  Support  and  encourage  project  managers’  interaction  with  Center  processes 

•  Lessen  the  burdens  on  branch  heads  and  business  financial  managers  by  facilitating  project 
managers’  access  to  quality  information  sources 

•  Ease  the  burdens  on  both  project  managers  and  leadership  by  designing  affordances  to  leverage 
existing  information 

•  Optimize  existing  information  to  be  easier  to  understand  such  as  status  reports,  data  calls  and 
other  required  documentation 

•  Add  value  through  strategic  content  location 

•  Improve  professionalism  and  ability  to  learn  with  consistent  stylized  icons  and  symbols 

•  Generate  consistency  across  navigation  elements,  interaction  behaviors,  and  language 

These  recommendations  were  all  based  on  initial  user  research,  knowledge  elicitation  sessions,  and 
heuristic  reviews  of  the  initial  portal  design.  Further  information  on  this  round  of  testing  is  in 
Appendix  A. 

4.2  APRIL  TEST 

Wireframe  usability  tests  conducted  in  April  revealed  that  the  portal  interface  design  and  planned 
interaction  models  were  increasingly  successful.  Users  were  delighted  with  the  portal’s  look  and  feel 
and  felt  optimistic  that  could  perform  project  management  tasks  through  the  interface.  The  ongoing 
challenge  is  to  design  an  interface  that  co-located  all  tools  and  resources  required  for  project 
management  in  an  easily  understood  and  navigable  format. 

The  resultant  recommended  design  changes  from  the  UCD&E  team  focused  on  the  following: 

•  One-stop  Shopping:  Decrease  user  workload  and  frustration  while  increasing  buy-in  and 
participation  by  providing  a  single  source  to  complete  desired  tasks 

•  Navigation:  Provide  users  with  a  simple,  clear  navigation  experience 
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The  resultant  recommended  design  changes  continued: 

•  Plain  Language:  Use  clear  and  consistent  terminology  to  increase  recognition,  ability  to  learn, 
and  user  trust 

•  Support  User  in  Task  Completion:  Ensure  that  the  system  and  interface  take  the  burden  for 
information  recall  and  task  recognition  off  the  user 

•  Increase  Interface  Transparency  and  Communication:  Tailor  the  interface  to  communicate 
value  specific  to  the  user. 

Through  the  UCD&E  team’s  iterative  design-test-revise-develop  model,  design  updates  were 
planned,  recommended,  and  mostly  implemented  as  the  portal  graduated  from  low-fidelity  wireframe 
testing  to  live  website -based  capabilities.  The  team  saw  continual  benefits  as  it  helped  identify  and 
isolate  user  expectations  and  flaws  in  the  portal  design.  Additionally,  as  the  product  team  and 
development  team  adopted  the  UCD&E  team’s  recommendations,  the  Center’s  user  community 
ultimately  received  a  more  consistent  and  professional  user  experience.  More  information  about  the 
results  of  the  April  testing  is  located  in  Appendix  B . 

4.3  MAY  TEST 

Usability  tests  of  early  portal  iterations  conducted  in  May  continued  to  show  successes,  especially 
navigation,  interface  design  and  user  task  performance.  Users  continued  to  give  overwhelmingly 
positive  feedback  about  the  look  and  feel  of  the  portal  and  demonstrated  that  they  could  perform 
basic  initial  project  management  tasks  through  the  interface  with  few  issues.  Priorities  developed 
from  the  results  of  the  May  tests  included  the  following: 

•  Removing,  de-emphasizing,  or  reducing  distractions  caused  by  project  management  body  of 
knowledge  (PMBOK)  Activities  and  Process  Framework  Overview 

•  Improving  placement  and  timeliness  of  feedback  and  feed-forward  mechanisms 

•  Using  clear  and  consistent  terminology  to  increase  recognition,  ability  to  learn,  and  user  trust 

•  Correcting  bugs  and  idiosyncrasies  in  the  website  design,  as  detailed  in  the  May  report 

Documented  results  were  compiled  and  reported  to  the  product  and  development  teams  as  part  of  a 
larger  report,  and  as  a  set  of  design  documents  that  pointed  out  each  feature  of  the  portal,  along  with 
the  UCD&E  team’s  recommendations.  These  recommendations  became  part  of  the  continuously 
updated  set  of  design  documents  maintained  by  UCD&E  and  communicated  to  the  development  team 
in  weekly  meetings  and  other  intermittent  interactions.  More  information  about  the  results  of  the  May 
testing  is  located  in  Appendix  C. 

4.4  A/B  STUDY 

At  the  same  time,  the  User-Centered  Design  and  Engineering  team  used  an  eight-question  survey 
to  compare  four  competing  design  options  to  understand  which  of  four  methods  of  secondary  portal 
navigation  users  deemed  “easier  to  use.”  The  result  of  the  survey  showed  two  choices  testing 
superior  to  the  rest.  Heuristic  evaluations  of  these  two  designs  concluded  that  one  was  likely 
insufficient  to  support  the  textual  density  of  the  content.  As  a  result,  the  project  Activity  navigation 
scheme  was  redesigned  as  part  of  the  left-hand  navigation  menu  to  match  the  results  from  this  A/B 
study. 
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Based  on  the  findings  of  these  studies,  it  became  evident  that  the  majority  of  major  issues 
impeding  user  performance  in  the  portal  had  been  identified.  Planned  resolution  steps  were  put  in 
place.  At  the  same  time,  the  development  team  was  beginning  heavy  development  work  to  match  the 
designs  proposed  by  the  UCD&E  team.  Due  to  these  factors,  traditional  user  testing  methodologies 
performed  previously  were  foregone  to  adopt  a  leaner  approach  designed  to  more  quickly  and 
effectively  identify  and  correct  portal  issues.  For  more  information  on  the  May  user  tests  and  A/B 
study,  see  Appendix  C. 

4.5  RAPID  USER  TESTS 

Rapid  User  Testing  is  a  tool  used  often  in  UCD&E  in  situations  where  a  research  team  wants  to 
speed  up  the  iterative  process  of  “test-design-code-repeat.”  In  Rapid  User  Testing,  changes  to  the 
user  interface  are  made  as  soon  as  a  problem  is  identified  and  a  solution  is  clear.  These  changes  can 
usually  be  done  in  a  single  day  with  minimal  investment  in  time,  equipment,  or  user  participation. 
Rapid  User  Testing  was  particularly  useful  for  this  project,  as  the  UCD&E  team  supported  the  rapid 
development  schedule  employed  by  the  development  team  to  successfully  meet  tight  schedule 
demands. 

As  part  of  the  effort  in  supporting  the  SSC  Pacific  project  portal,  the  UCD&E  team  conducted 
three  rounds  of  Rapid  User  Testing  to  quickly  and  efficiently  identify  and  correct  the  major  interface 
impediments  and  system  bugs  in  the  existing  design.  As  part  of  this  action,  several  key  interface 
elements  and  system  bugs  were  identified,  documented,  and  corrected.  The  most  significant  issues  to 
be  addressed  and  corrected  include  the  following: 

•  Shortening  computer  wait  times  between  tasks 

•  Correcting  issues  with  the  personnel  lookup  tool 

•  Debugging  and  implementing  fixes  for  several  bugs  discovered  during  testing 

The  vast  majority  of  elements  the  UCD&E  team  tested,  beyond  those  issues  documented  here, 
tested  extremely  well.  Users  successfully  completed  all  tasks  given  them,  apart  from  those 
exceptions. 

Overall,  the  three  rounds  of  user  testing  proved  to  be  sufficient  to  capture  the  large  majority  of  user 
experience  issues  in  what  the  UCD&E  team  could  test.  The  pace  was  fast  enough  to  prevent  the 
development  team  from  doing  unnecessary  rework  in  most  cases,  but  moderate  enough  to  be 
successful  with  a  smaller  research  staff.  The  UCD&E  team  would  have  ultimately  liked  to  do  more 
testing  on  other  areas  of  the  design,  but  due  to  schedule  constraints  from  management,  and  personnel 
constraints  in  the  development  team,  this  became  unrealistic.  For  more  information  on  the  Rapid 
User  Testing  performed  by  the  UCD&E  team,  please  see  Appendix  D. 
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5.  CONCLUSION 


Initial  user  research  into  the  SSC  Pacific  Project  Management  Portal  surfaced  a  strong  need  for  a 
clear  value  proposition  for  the  project  portal.  As  a  result,  it  is  critical  to  communicate  to  all  users  that 
the  project  portal  focuses  solely  on  the  Center’s  project  management  processes.  To  bring  value  to 
these  key  users,  co-locating  tools  and  resources  a  project  manager  will  need  to  accomplish  Center- 
level  project  management  process  tasks  is  essential.  Project  managers’  primary  focus  is  on  meeting 
project  needs  for  performance,  finance,  and  schedule.  This  need,  and  the  importance  of  attending  to 
the  priorities  of  their  sponsors,  requires  that  the  project  portal  deliver  real  value,  and  not  become  just 
demand  compliance.  The  portal’s  value  can  only  be  fully  appreciated  when  it  seamlessly  becomes 
part  of  a  project  manager’s  existing  activities,  rather  than  just  one  more  onerous  task  that  must  be 
endured. 

Usability  test  results  revealed  that  the  interface  design  and  the  planned  interaction  models  were 
highly  successful.  Users  were  delighted  with  the  look  and  feel  and  felt  optimistic  that  they  could 
perform  their  project  management  tasks  through  the  interface.  Virtually  everything  the  UCD&E  team 
tested,  beyond  issues  documented  in  this  test  report,  tested  extremely  well.  Users  successfully 
completed  the  tasks  tested  in  any  given  round,  apart  from  these  exceptions. 

Test  results  show  the  portal  design  is  easy  to  understand,  making  repeated  tasks  easy  to  learn  and 
remember  for  participants.  As  each  participant  repeated  tasks,  the  time  and  effort  required  to 
complete  the  task  was  reduced  with  each  repetition.  Repeated  performances  resulted  in  easier  and 
quicker  task  completion.  This  result  is  a  key  finding  and  a  major  success  as  well.  The  portal  design 
enables  users  to  quickly  become  “experts”  in  performing  basic  tasks  after  two  or  three  instances.  This 
finding  suggests  that  the  site,  as  designed,  will  lead  to  improved  experience  and  greater  performance 
capabilities  among  most  of  the  user  population. 

The  UCD&E  team  concludes  that  the  amount  of  user  testing  was  sufficient  to  capture  the  large 
majority  of  user  experience  issues  in  what  the  UCD&E  team  had  available  to  test.  Later-developed 
functionality  was  not  user-tested  in  an  active  environment  due  to  schedule  and  development  issues. 
Generally,  the  pace  of  testing  was  fast  enough  to  keep  the  development  team  from  doing  unnecessary 
rework,  but  moderate  enough  to  be  performed  with  a  moderate-sized  research  team. 

Although  the  overall  design  tested  very  well,  flaws  associated  with  the  use  of  plain  language  and 
clear  context  were  observed  in  various  elements  of  the  portal  design.  Additionally,  continued  user 
research  through  multiple  rounds  of  usability  tests  highlighted  the  need  for  a  strong  and  clear  value 
proposition  for  the  product.  The  UCD&E  team  corrected  these  flaws  as  part  of  its  final  design 
recommendations,  but  any  additions  in  language  or  functionality  not  included  in  those  designs  is  in 
danger  of  eroding  and  damaging  the  value  proposition  presented  to  the  users. 

Finally,  as  previously  noted,  only  providing  users  a  product  focused  solely  on  Center  project 
management  processes  brings  little  to  no  value  to  targeted  users  within  the  project  manager  and 
Integrated  Product  Team  (IPT)  lead  population  segments.  Unless  the  portal  can  provide  users  a 
universally  delightful  experience  and  a  non-negative  value  to  their  project  work,  participation  will 
struggle  and  adoption  will  lag  to  the  point  that  the  portal  will  bring  little  benefit  to  the  Center.  The 
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UCD&E  team  is  confident  that  its  iterative  design-test-revise-develop  testing  model  has  produced 
many  benefits  as  it  helped  reveal  user  expectations  and  flaws  in  webpage  designs.  Additionally,  with 
continued  attention  to  UCD&E’s  design  recommendations,  the  Center’s  user  community  will 
ultimately  be  well  served  with  a  consistent  and  professional  user  experience. 
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6.  LESSONS  LEARNED 


As  team  members  and  collaborators  in  this  novel  but  important  project,  the  UCD&E  team  has 
discovered  five  lessons  learned  from  its  participation.  These  are  documented  to  track  and  improve 
upon  UCD&E  performance  in  any  further  iterations  of  this  project  and  to  further  promote 
professionalism  in  other  project  relationships. 

1 .  A  clear  communication  path  of  agreed  communication  methods  needs  to  be  established  and 
used  for  tracking  and  resolving  issues  as  they  occur.  Communications  issues  arose  when 
dealing  with  accepting  and  disseminating  documents  with  other  stakeholders.  Previously, 
the  UCD&E  team  had  a  preferred  method  of  internal  communication  of  test  results, 
PowerPoint  slides,  Excel®  and  design  documents,  wireframes,  etc.  using  a  configuration 
management  system  meaningful  to  the  team.  In  this  project,  other  team  stakeholders  used 
other  methods  (e.g.,  JIRA)  to  communicate,  track  bugs  and  design  issues,  and  manage  the 
project.  In  the  future,  the  UCD&E  team  needs  to  be  especially  clear  upfront  about  how  and 
when  to  participate  in  these  tracking  systems,  especially  when  communicating  design 
changes  and  deliveries.  To  avoid  confusion,  all  team  participants  need  to  be  using  the  same 
management,  bug  tracking,  and  communication  methods. 

2.  The  UCD&E  team  should  avoid  communicating  directly  with  other  teams  using  low- 
fidelity  wireframes,  unless  it  is  a  critical,  emergent  circumstance  that  necessitates  sharing 
them  with  this  larger  audience.  After  delivery  additional  communication  issues  arose 
particularly  relating  to  design  sketches.  There  were  some  miscommunications  and 
assumptions  made  about  low-fidelity  sketches  when  they  were  disseminated  to  the 
development  and  management  teams,  which  caused  unneeded  issues  during  subsequent 
interactions.  Communication  through  high-fidelity  prototypes  is  more  helpful  in  accurately 
demonstrating  the  design’s  interaction  and  intent  with  both  stakeholders  and  developers. 

3.  A  detailed  explanation  of  the  role  and  importance  of  UCD&E  methods  is  needed.  This 
explanation  would  detail  the  practice  and  implications  of  good  user-centered  design  is  an 
ongoing  process.  Despite  many  customers,  stakeholders,  and  other  value-chain  partners 
realizing  the  need  for  proper  UCD&E  in  their  products’  design  cycle,  the  UCD&E  team 
cannot  assume  that  they  appreciate  or  understand  the  complexities  and  nuances  of  how  or 
why  this  effort  is  performed.  In  the  future,  the  UCD&E  team  needs  to  better  familiarize  all 
parties  with  UCD  processes  and  terminology,  encouraging  their  appreciation  of  human 
factors  and  UCD  efforts.  Through  consistent  communication  and  instruction  about  UCD 
processes,  benefits,  and  results  to  all  parties,  we  can  engender  real  understanding,  and 
avoid  many  misunderstandings  and  disputes. 

4.  The  timeline  presented  initial  problems  for  the  UCD&E  team.  Because  of  uncertainty 
surrounding  the  level  of  effort  being  provided  to  the  project  by  the  UCD&E  team,  there 
were  a  few  speed  bumps  in  getting  the  team  sufficiently  tasked  and  manned  in  the 
beginning.  In  the  future,  a  fully  successful  integration  of  the  agile  development  and  user- 
centered  design  processes  requires  a  more  formal  ramping  up  stage  (i.e.,  a  “Sprint  Zero”) 
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focused  on  user  research  and  initial  design.  Because  this  work  was  accomplished  in 
parallel  with  the  development  of  the  site,  it  was  initially  difficult  to  get  far  enough  ahead  of 
the  development  process  to  sufficiently  implement  all  of  the  UCD&E  team’s  methods  and 
best  practices  for  conducting  proper  focused  user  testing  ahead  of  implementation  by  the 
development  team. 

5.  Managing  the  expectations  of  the  user  population  is  extremely  important  to  the  ultimate 
goal.  During  testing  cycles,  there  are  intermittent  periods  where  the  UCD&E  team  asks 
open-ended  questions  of  the  test  participants.  Some  of  these  questions  focus  on  user 
desires,  preferences,  needs,  likes,  and  expectations.  From  the  user  response,  the  team  can 
determine  which  functionalities  and  features  are  most  important  to  include,  how  they 
should  be  structured,  best  use  of  navigation  elements,  etc.  During  this  process,  it  became 
evident  that  a  couple  things  were  happening  simultaneously.  Test  participants  were 
developing  unrealistic  expectations  of  what  the  portal  would  ultimately  provide.  Also, 
users’  preferences  and  needs  diverged  greatly  from  what  stakeholders  had  asked  to  be 
developed.  Finally,  some  users  at  the  Center  were  being  given  information  about  the  portal 
and  its  eventual  capabilities  before  the  design  was  finished. 

These  issues  were  counterproductive  for  a  couple  of  reasons.  Because  of  the  intimate  nature  of  the 
close-knit  groups  and  the  ability  for  information  to  spread  quickly  at  the  Center,  false  information 
can  greatly  affect  the  expectations  and  rate  of  adoption  of  these  future  users  if  the  product  doesn’t 
meet  those  expectations.  Additionally,  if  users  have  preconceived  notions  about  the  final  product 
before  they’re  tested,  those  biases  can  creep  into  the  findings,  thereby  affecting  the  overall  results. 
Likewise,  if  users  are  influencing  the  design  of  the  final  product,  it  inhibits  the  UCD&E  team’s 
ability  to  create  the  final  design  to  account  for  the  entire  tested  user  population. 

As  a  result,  future  efforts  need  to  control  more  tightly  the  information  flow  regarding  the  purpose 
and  goals  of  the  product  being  designed.  At  times,  the  message  simply  outpaced  the  design,  and  that 
seems  to  have  set  up  some  issues  of  confusion  and  mistrust  among  some  of  the  ultimate  user 
population.  Through  future  efforts  to  manage  the  value  proposition  of  the  product  at  all  stages  of 
development,  the  UCD&E  team  can  avoid  delivering  a  product  that  is  not  in  line  with  users’ 
expectations,  thereby  slowing  adoption  and  forfeiting  trust. 
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APPENDIX  A 


EXECUTIVE  SUMMARY 

Initial  user  research  into  the  project  management  portal  indicates  a  fundamental  need  for  clarity 
across  the  product  interface  to  address  shortfalls  with  the  overall  content,  goals,  and  value  to  the  user. 
The  team’s  focus  moving  forward  should  remain  on  the  product’s  goal  of  becoming  the  single 
interactive  source  for  all  Center  project  management  processes. 

Participants  said  the  following: 

“The  project  management  guide  is  found  through  a  serious  of  search  and  rescue  steps  on 
corporate  assets  to  find  the  current  version.” 

Future  designs,  recommendations,  and  briefings  by  the  User  Centered  Design  and  Engineering 
(UCD&E)  team  will  address  the  following: 

1 .  Incorrect  assumptions  about  project  managers  at  the  Center 

2.  Lack  of  clarity  across  the  product  regarding  its  content,  goals,  and  value 

3.  Improved  focus  on  Center  product  management  processes  instead  of  day-to-day  project 
management  tools 

4.  Supporting  and  encouraging  proactive  interaction  with  Center  processes  by  project 
managers 

5.  Easing  the  burdens  on  critical  players,  such  as  branch  heads  and  business  financial 
managers,  by  providing  their  project  managers  with  alternative,  quality  information 
sources 

6.  Easing  the  burdens  on  both  project  managers  and  leadership  by  designing  for  information 
roll-up  and  leveraging  existing  information  for  easier  and  more  efficient  status  reports,  data 
calls,  and  other  required  documentation 

7.  Adding  value  by  using  strategic  content  placement 

8.  Improving  the  professionalism  and  ability  to  learn  and  ease  of  use  of  the  interface  with 
custom  icons  and  symbols 

9.  Continuing  to  address  consistency  across  navigation  elements,  interaction  behaviors,  and 
language 

These  user-centered  design  recommendations  are  based  on  initial  user  research  and  heuristic 
reviews  of  the  product’s  goals  and  requirements.  Additional  user  research,  including  usability  testing, 
will  modify  and  contribute  to  these  findings. 
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PRIMARY  ACTION  ITEMS 


OVERVIEW 

This  section  contains  a  listing  of  action  items  that  were  found  during  the  early  stage  of  portal 
research.  Table  A-l  shows  global  featured  action  items  uncovered  during  the  initial  user  research. 
Additional  detail  about  each  of  these  items  is  in  the  Findings  Section. 


Table  A-1.  Global  featured  action  items  uncovered  during  initial  user  research. 


Topic 

Action  Item 

Priority 

Alter  the  value  proposition: 
language  used  throughout 
the  portal  is  incongruent 
with  users’  mental  models 

Increase  the  value  proposition  of  the  product  by  defining  its  role 
as  a  single  source  for  Center  project  management  processes. 

Use  consistent  and  clear  language  to  differentiate  this 
“reference  library”  from  a  set  of  day-to-day  project  management 
tools. 

High 

Remove  assumptions: 
users  do  not  share  a 
universal  understanding  of 
the  Center’s  project 
management  processes 

Incorporate  self-directed  learning  opportunities  into  the  interface 
to  account  for  users  with  little  to  no  awareness  of  the  Center’s 
project  management  practices. 

High 

Remove  assumptions:  not 
all  users  proactively 
complete  Center-level 
project  management 
processes 

Communicate  Center-level  process  tasks  and  deadlines  directly 
to  the  project  managers,  bypassing  the  branch  heads. 

High 

Change  the  primary 
information  source: 

remove  the  burden  from 
the  portal  user 

A)  Build  user’s  trust  by  ensuring  easy  access  to  current 
information  whenever  a  user  accesses  the  product. 

B)  Emphasize  to  branch  heads  and  other  support  staff  that  this 
product  will  relieve  them  of  the  burden  of  the  “middleman” 
in  their  relationships  with  project  managers. 

High 

Create  efficiency  by 
rolling-up:  simplify 
interfacing  with  leadership 

Keep  all  project  content  visible  and  exportable  by  the  project 
manager.  Content  should  pre-populate  (cross-pollinate)  across 
and  within  all  Center  project  management  processes  and  tools. 

Medium 

Separate  disparate 
elements:  differentiate 
between  day-to-day 
project  management  tools 
and  Center-level  project 
management  processes 

Focus  initial  increments  on  only  managing  Center-level  project 
management  processes.  Avoid  forcing  changes  to  project 
manager’s  day-to-day  methods  for  monitoring  finances, 
performance,  and  scheduling. 

Medium 
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Table  A-1 .  Global  featured  action  items  uncovered  during  initial  user  research  (continued). 


Topic 

Action  Item 

Priority 

Make  calendar  items 
interoperable 

Translate  process  deadlines  into  formats  typically  used  by 
project  managers  to  export  from  the  portal,  including  but  not 
limited  to:  GANTT  charts,  Microsoft  Outlook®  calendar  items 
(desktop  and  web  compatible),  CSV  tables,  Microsoft  Visio®, 
and  Microsoft  Project®.  Focus  on  interoperability  across 
multiple  platforms;  (do  not  make  assumptions  about  which 
software  users  prefer). 

Medium 

Increase  ability  to  learn 
and  efficiency  with 
consistent  semiology 

Evaluate  the  symbol  libraries  already  available  for  the  Center 
and  hire  a  graphic  designer  to  help  customize  and  unify  the 
icons  for  the  different  product  elements. 

Medium 

Add  value  through 

content:  team 

management 

Investigate  surfacing  information  from  Navy  ERP  about  who  is 
charging  to  a  project,  and  co-locating  other  information  about 
team  members  inside  the  product. 

Low 

Add  value  through 
content:  financial 

Investigate  automatically  tracking  financial  information  that  the 
business  financial  managers  already  supply  and  display  it  within 
the  product  to  provide  visibility  to  project  managers  about  their 
projects’  expected  spending,  planned  spending,  and  actual 
spending  rates. 

Low 

Add  value  through 
content:  risks 

Investigate  surfacing  risk  information  from  its  current  project 
management  tool,  and  other  risk-tracking  methods  for  potential 
interoperability  for  visibility  by  project  managers. 

Low 

Additional  Action  Items  that  surfaced  during  the  user  research  and  heuristic  reviews  and  which  the 
UCD&E  team  could  address  through  their  design  process  have  been  incorporated  into  the  baseline 
design.  The  issues  that  were  included  inconsistent  use  of  language,  problems  with  the  navigational 
structure,  and  alignment  problems. 

MISSING  OR  REQUESTED  CONTENT  AND  FEATURES: 

•  Risks 

•  Financial  view  -  line  graph  with  the  expected  spending  curve,  planned  spending  curve,  and 
actual  spending  curve 

•  The  project  management  plan 

•  Gantt  charts 

•  Customization 

•  Options  for  archiving  content  and  submitted  documents 

•  Tracking  for  projects  in  the  proposal  phase  prior  to  initiation 
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PRODUCT  AND  DESIGN  GOAL 


OVERVIEW 

This  section  contains  product  and  design  goals  that  were  used  in  the  portal  evaluation  study.  The 
goal  of  this  initial  product  is  to  provide  a  single  space  for  Space  and  Naval  Warfare  Systems  Center 
(SSC)  Pacific’s  project  managers  to  access  any  type  of  Center  project  management  content  necessary 
for  their  project.  Additionally,  the  space  is  expected  to  provide  customized  project  management 
assistance  based  on  the  characteristics  of  the  projects  a  project  manager  is  handling. 

GOAL  OF  THIS  RESEARCH 

Research  focused  on  the  initial  structural  and  content  requirements  for  the  product  (a.k.a.  portal). 
Navigation  elements,  consistency  in  language  and  shared  understanding  of  the  language  used,  and  the 
overall  functional  requirements  of  project  managers  formed  the  basis  of  the  research. 

ASSUMPTIONS 

The  portal  is  a  project  management  tool,  accessed  using  CAC  identification  and  specific  to  the 
projects  a  project  manager  is  working  on.  Users  have  some  background  or  training  in  project 
management  processes  as  practiced  at  Space  and  Naval  Warefare  Systems  Pacific. 
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USERS 


OVERVIEW 

This  section  contains  an  overview  of  the  background  of  the  participants  for  the  portal  study.  This 
initial  research  involved  10  30-minute  participant  interviews. 

Participants  had  the  following  dimensions: 

•  Gender:  both  male  and  female 

•  Experience:  less  than  a  year  up  to  16  plus  years 

•  Number  of  concurrent  projects:  from  1-1 1  plus,  with  most  managing  1-3 

•  Codes:  H000M,  84100,  72120,  63200,  56442,  56441,  56230,  56220,  53223,  52610 

The  chart  shown  in  Figure  A-l  details  the  top  two  sources  of  information  about  project 
management.  Participants  were  presented  with  two  randomly  presented  options.  Participants  selected 
the  “Switchboard”  and  “Colleagues”  as  their  primary  sources  for  information  about  project 
management.  Results  in  the  “Other”  category  included  “Project  Manager  Box,”  “Division  Project 
Manager  Meetings,”  and  “Other  Project  Manager  Sites.” 


Figure  A-1 .  Information  sources  for  participant  project  managers. 
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METHODOLOGIES 


OVERVIEW 

This  section  contains  three  methods  that  were  used  for  the  portal  study.  The  initial  round  of  user 
research  included  a  semi-structured  interview  and  an  initial  interface  walkthrough,  which  took 
more  than  30  minutes.  The  goal  of  these  methodologies  was  to  elicit  user  knowledge  about  the 
domain  and  to  discover  task  content. 

Step  1:  Pre-Interview  Data  Collection 

Prior  to  the  user  interviews,  participants  filled  out  an  online  form  with  basic  questions  concerning 
the  amount  of  time  they’ve  been  a  project  manager,  the  number  of  projects  they  typically  manage, 
and  their  typical  sources  of  project  management  information. 

Step  2:  Semi-Structured  Interview 

The  semi-structured  interview  included  specific  scripted  questions  with  the  addition  of  optional 
probing  questions  (e.g.,  “Tell  me  more...”).  The  questions  allowed  participants  to  elaborate  on  their 
answers.  This  interview  was  designed  to  capture  basic  information  about  the  participants’  project 
management  priorities,  daily  activities,  tools,  questions,  and  general  methodologies. 

Step  3:  Initial  Interface  Guided  Walkthrough 

The  guided  walkthrough  was  designed  to  step  participants  through  low-fidelity  wireframes  to 
observe  their  initial  reactions,  and  examine  how  well  their  expectations  of  the  interface  were  met. 
Additional  goals  included  understanding  participants’  use  of  navigational  elements,  their  ability  to 
recognize  the  content  presented,  and  evaluation  the  overall  usefulness  of  the  interface.  This  part  of 
the  research  was  also  conducted  in  a  semi-structured  manner,  giving  the  moderator  a  chance  to 
venture  off-script,  depending  on  participant’s  responses. 
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FINDINGS 


OVERVIEW 

This  section  contains  findings  found  during  all  parts  of  the  study  for  this  report.  In  this  Section, 
findings  are  briefly  described  with  a  summary  of  the  finding.  Comments  from  portal  users  specific  to 
issues  shown,  the  potential  impact  of  the  issue  is  defined,  and  an  action  item  is  given  of  plans  to 
correct  the  issue. 

ALTER  THE  VALUE  PROPOSITION:  LANGUAGE  USED  THROUGHOUT  THE  PORTAL  IS 
INCONGRUENT  WITH  USERS’  MENTAL  MODELS 

Summary 

Value  proposition  concerns  what  users  think  they  can  do  with  a  product  or  in  a  space  based  on 
multiple  interface  elements,  including  but  not  limited  to:  language,  navigation,  look  and  feel,  and 
affordances.  This  product  created  confusion  among  the  participants  as  to  its  actual  value  proposition. 
The  participants’  confusion  was  based  primarily  on  language  and  content  issues.  They  were  unclear 
as  to  which  aspects  of  project  management  the  product  was  addressing.  When  the  researchers  asked 
for  further  clarification,  participants  explained  that  they  think  differently  about  day-to-day  project 
management  practices  and  the  processes  they  use  to  satisfy  Center-level  project  management 
requirements. 

Our  study  found  participants  value  propositions  are  not  addressed  through  the  portal  as  presently 
designed: 

•  “Project  management”  and  “project  management  tools”  refer  to  the  day-to-day  activities  and 
supporting  software  or  tools  required  to  make  sure  a  team  and  or  project  is  running  smoothly, 
project  managers  tailor  these  activities  to  their  team,  project  type,  and  environment  while 
executing  their  projects  using  multiple  tools. 

•  “Center-level  project  management  processes”  is  conceptualized,  not  as  day-to-day  activities 
bringing  value  to  the  fast-paced,  real-time  world  of  project  management,  but  as 
Center-mandated  additions  to  their  project's  existing  project  management  lifecycle.  Project 
managers  expect  to  check  these  processes  rarely,  as  they  occur  only  at  single  points  in  the 
project  Management  lifecycle. 

Participants  said  the  following  about  the  portal  compatible  preconceived  perceived  mental 
models: 

“. .  .to  me  it  looks  like  what  upper  management  wants  for  a  warm  and  fuzzy.  Doing  this 
doesn't  mean  you're  managing  well.  Do  we  want  it  to  be  how  they  actually  manage  their 
projects  or  just  the  information  that  is  needed  to  push  up  the  chain?” 

Participants  specified  that  since  these  two  types  of  activities  occur  during  different  periods,  they 
prefer  not  to  manage  them  in  the  same  spaces. 
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Impact 

The  impact  of  this  distinction  is  that  multiple  participants  did  not  see  the  value  of  placing 
day-to-day  administrative  information  and  Center-mandated  process-related  information  side  by  side. 
Participants  did  see  the  value  of  single  source  access,  but  stated  they  would  not  be  accessing  it 
frequently. 

Participants  said  the  following: 

“This  looks  like  something  the  Center  would  create.  And  what  I  mean  by  that  is  that  I  would 
view  most  of  the  bottom  half  [process  phases]  as  overhead  and  not  valuable  to  me . . .  [but] ...  it 
would  be  easier  to  know  what  was  expected  of  me  and  then  manage  to  it.  If  that  is  the  intent, 
then  this  would  be  helpful.” 

All  participants  used  different  techniques  and  software  for  managing  day-to-day  practices; 
moreover,  most  of  these  were  specific  to  the  type  of  project  or  product  managed.  Participants  view 
those  tools  and  practices  separately  from  the  Center’s  project  management  processes. 

Action  Item 

Increase  the  value  proposition  of  the  product  by  defining  its  role  as  a  single -source  entity  for 
Center  project  management  processes.  Use  consistent  and  clear  language  to  differentiate  the  portal, 
providing  process  information  from  day-to-day  project  management  tools. 

REMOVE  ASSUMPTIONS:  USERS  DO  NOT  SHARE  A  UNIVERSAL  UNDERSTANDING  OF  THE 
CENTER’S  PROJECT  MANAGEMENT  PROCESSES 

Summary 

The  interviews  surfaced  a  range  of  perceptions  from  project  managers  about  the  actual  processes 
required  by  the  Center,  the  tools  that  support  those  processes,  and  the  phases  of  project  management 
(i.e,  initiation,  planning,  executing,  monitoring,  and  closing).  Some  participants  claimed  that  they  are 
educating  themselves  on  these  topics  using  project  management  body  of  knowledge  resources  to 
support  their  independent  learning  activities,  but  most  of  the  participants  were  unfamiliar  with  these 
phases. 

A  project  manager  with  one  to  five  years’  experience  handling  two  to  three  projects  concurrently  said 
the  following: 

“I'm  confused  by  the  different  phases  [initiation,  planning,  execution,  monitoring,  closing] 
the  whole  part  of  the  bottom  I  don't  know  what  I  would  do  there;  I  don't  know  what  that 
would  be.” 

Impact 

This  indicates  that  our  assumption  that  project  managers  have  a  basic  working  knowledge  of  the 
Center’s  project  management  processes  is  false. 


A-1 2 


Action  Item 


Incorporate  self-directed  learning  opportunities  into  the  interface  to  account  for  users  with  little  to 
no  awareness  of  the  Center’s  project  management  practices. 

REMOVE  ASSUMPTIONS:  NOT  ALL  USERS  PROACTIVELY  COMPLETE  CENTER-LEVEL 
PROJECT  MANAGEMENT  PROCESSES 

Summary 

Not  all  project  managers  are  aware  of  or  current  with  all  crucial  action  items  or  due  dates  for 
Center-level  processes.  In  some  cases,  responsibility  for  these  actions  seems  to  fall  on  branch  heads 
or  overarching  project  managers  who  then  push  the  requirements  to  the  day-to-day  project  managers. 
This  is  also  producing  an  “eleventh-hour  effect,”  with  project  managers  stating  that  they  wait  until 
the  last  minute  to  complete  process  tasks  because  being  proactive  is  not  perceived  as  adding  value. 

The  following  comment  was  given  concerning  the  Center-level  project  management  processes: 

“Basically  when  my  branch  head  emails  me  and  tells  me  I  have  to  do  this,  I  wait  until  I'm 
going  to  get  into  trouble  if  I  don't  do  it.  What  I  found  is  if  I  try  to  be  on  top  of  it  and  be 
proactive,  it  takes  so  much  energy  to  figure  out  what  [the  process  requirements]  are,  they  will 
probably  change  by  the  time  I've  figured  it  out.  By  the  end,  I  will  have  done  more  work  than  I 
had  to,  too  much  work,  and  it  wasn't  helpful  to  my  project.” 

Impact 

Project  managers  are  pushing  deadlines  for  Center-level  processes,  creating  risk  of  an  overdue 
process,  and  creating  more  work  for  branch  heads  by  shifting  the  burden  of  responsibility  onto  them 
for  on-time  notification  and  tasking  about  Center-level  processes. 

Action  Item 

Communicate  Center-level  process  tasks  and  deadlines  directly  to  the  project  managers,  bypassing 
the  branch  head. 

CHANGE  THE  PRIMARY  INFORMATION  SOURCE:  REMOVE  THE  BURDEN  FROM  THE 
PEOPLE 

Summary 

Most  of  the  participants  see  one  or  more  colleagues  as  primary  sources  of  information  about 
project  management.  Typically,  the  branch  head  was  included  as  the  “go-to-person”  for  information 
about  any  process. 
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Participants  said  the  following: 

“I  will  contact  one  of  two  people  in  my  branch  if  I  need  assistance:  the  branch  head  and 
another  project  manager.” 

Impact 

There  is  a  potential  for  misinformation  when  most  communications  are  not  centrally  located  and 
shared  verbally.  However,  this  strong  cultural  trend  of  reliance  on  word-of-mouth  assessments  is 
leveraged  to  develop  trust  in  the  new  interface  if  the  interface  is  well  received  in  the  first  place.  If 
not,  there  will  be  huge  pushback  at  multiple  levels  and  user  trust  will  fail  to  form. 

Action  items 

1.  Build  users’  trust  by  ensuring  easy  access  and  current  information  whenever  a  user 
accesses  the  product. 

2.  Emphasize  to  branch  heads  and  other  supporting  staff  that  this  product  will  relieve  them  of 
the  burden  of  the  “middleman”  in  their  relationships  with  project  managers. 

CREATE  EFFICIENCY  BY  ROLLING-UP:  SIMPLIFY  INTERFACING  WITH  LEADERSHIP 
Summary 

Most  users  are  looking  for  faster  and  easier  ways  to  submit  status  reports,  satisfy  data  calls,  and 
create  presentations.  Participants  were  hopeful  that  the  portal,  as  a  centralized  source  of  process 
information,  including  all  the  information  they  are  already  entering  into  tools  like  the  Work 
Acceptance  tool  (WAT),  would  leverage  their  data  for  new  items  such  as  status  reports. 

Participants  said  the  following: 

.Power  Point,  the  Center’s  format  for  project  reviews.  The  quad  charts  should  not  be 
generated  [by  the  project  manager],  but  derived  from  data  that  is  already  there.  The  quad 
chart  should  be  automatically  generated;  the  project  manager  shouldn’t  have  to  do  their  own. 
The  automation  should  be  in  work  acceptance,  and  if  it's  not  there,  then  add  it.  The  rest  of  the 
briefs  should  be  the  same  way;  we  shouldn't  be  extracting  data  from  this  tool.” 

Impact 

The  ability  to  re-use  previously  entered  data  to  meet  Center  project  management  needs  would  be  a 
large  benefit  in  time  and  efficiency  for  most  project  managers. 

Action  Item 

Keep  all  project  content  visible  and  exportable  by  the  project  manager.  Content  should  pre¬ 
populate  (cross-pollinate)  across  and  within  all  Center-level  project  management  processes  and  tools. 


A-14 


Participants  said  the  following: 

“The  data  calls  usually  come  with  very  little  time  so  they  get  partial  or  old  data,  not  as  good 
as  could  be  done  if  they  had  an  automated  way  of  pulling  and  maintaining.” 

SEPARATE  DISPARATE  ELEMENTS:  DIFFERENTIATE  BETWEEN  DAY-TO-DAY  PROJECT 
MANAGEMENT  TOOLS  AND  CENTER-LEVEL  PROJECT  MANAGEMENT  PROCESSES 

Summary 

Most  project  managers  stated  that  their  goal  in  day-to-day  project  management  is  tracking 
schedule,  performance,  and  critical  financial  elements.  They  use  a  multitude  of  resources  and 
methods,  usually  tailoring  each  one  to  fit  the  specific  needs  of  their  project.  For  example,  software 
development  project  managers  talked  about  using  backlogs  and  user  stories,  while  other  project 
managers  talked  about  installation  timelines.  However,  most  agree  that  some  form  of  this  information 
is  the  most  critical  to  understanding  the  real-time  health  and  status  of  their  projects.  Projects  have 
three  key  things:  financials,  performance,  and  schedules. 

Impact 

Without  total  interoperability  and  automated  data  pushes,  these  elements  could  quickly  go  stale  in 
the  product.  Implementing  interoperability  and  automated  data  pushes  will  be  a  significant  challenge, 
especially  for  those  managing  many  different  types  of  day-to-day  operations.  Some  of  this 
information  may  bring  value  to  the  project  management  interface,  but  is  also  likely  to  cause 
discontent  if  users  are  forced  to  abandon  their  established  methods  for  a  tool  that  is  not  tailored  to 
their  type  of  project  and  their  specific  day-to-day  project  management  needs. 

Action  Item 

Focus  initial  increments  on  only  managing  Center-level  project  management  processes.  Avoid 
forcing  changes  to  project  manager’s  day-to-day  methods  for  monitoring  finances,  performance,  and 
scheduling. 

MAKE  CALENDAR  ITEMS  INTEROPERABLE 
Summary 

Users  have  a  variety  of  different  methods  for  calendaring  or  tracking  their  action  items.  Most 
stated  that  an  additional  calendar  would  not  be  useful  and  could  quickly  become  stale.  However, 
knowing  the  required  deadlines  for  the  Center  processes  was  still  valuable  information. 

Sub-calendaring  note:  multiple  users  reported  using  a  paper-based  calendaring  system  in 
conjunction  with  their  online  tools. 

Participants  said  the  following: 

“The  calendar,  unless  it’s  the  best  calendar  on  the  face  of  the  earth  ...  I  already  have  too 
many  calendars” 
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Impact 

Participants  said  the  following  about  including  a  calendar  in  the  portal  interface: 

1 .  It  is  likely  only  a  small  percentage  of  users  will  use  it. 

2.  Items  are  likely  to  become  stale  and  not  accurately  reflect  the  project  to  leadership. 

3.  It  must  be  interoperable  for  multiple  platforms  and  reflect  different  methods  of  use. 

Action  Item 

Translate  project  management  process  deadlines  into  formats  typically  used  by  project  managers  to 
export  from  the  portal,  including  but  not  limited  to  GANT  charts,  Microsoft  Outlook®  calendar  items 
(desktop  and  web-compatible),  CSV  tables,  Visio®,  and  Microsoft  Project.  Focus  on  interoperability 
across  multiple  platforms;  do  not  make  assumptions  about  which  software  users  prefer. 

INCREASE  LEARNABILITY  AND  EFFICIENCY  WITH  CONSISTENT  SYMBOLOGY 
Summary 

To  increase  recognition,  ability  to  learn,  and  consistent  navigation,  the  portal  as  a  whole  needs  to 
use  consistent  symbology  that  is  minimal,  clear,  and  unambiguous.  Any  overuse  will  demand  greater 
cognitive  processing,  but  consistent  use  in  the  same  format  every  time  will  increase  trust  and  speed 
up  the  user’s  ability  to  navigate  the  portal.  The  symbol  library  designed  and  chosen  needs  to  be 
appropriate  to  SSC  Pacific,  follow  usability  rules,  and  have  a  professional  look  and  feel  that  matches 
the  overarching  theme. 

For  this  type  of  product,  the  UCD&E  team  recommends  hiring  a  graphic  designer  to  work  in 
conjunction  with  the  UCD&E  team. 

Participants  said  the  following: 

“...It’s  rudimentary;  the  appearance  of  the  icons  like  that  leads  me  to  believe  this  tool  might 
not  be  for  me.” 

Impact 

All  icons  and  symbols  across  the  suite  will  be  both  consistent  and  professional.  The  goal  is  to  have 
only  one  icon  for  every  unique  element  requiring  an  icon,  and  to  have  a  professional  look  and  feel 
that  inspires  trust. 

Action  item 

Evaluate  the  symbol  libraries  already  available  for  the  Center  and  hire  a  graphic  designer  to  help 
customize  and  unify  the  icons  for  the  different  product  elements. 
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ADD  VALUE  THROUGH  CONTENT:  TEAM  MANAGEMENT 


Summary 

Project  managers  expressed  an  interest  in  having  more  information  about  their  team  easily 
accessible  (e.g.,  not  in  Navy  ERP).  Content  of  interest  includes,  amount  charged  to  the  project,  total 
hours  charged,  hours  or  charge  amount  still  available,  contact  information,  and  group  messaging. 

Participants  also  noted  that  it  is  difficult  to  discover  the  name  of  every  person  who  is  charging  to 
the  project. 

Participants  said  the  following: 

“I  would  like  to  know  who  is  charging  to  those  efforts  . . .  there  are  miscellaneous  people  who 
charge  to  that  project.  It  is  another  feat  of  magic  through  ERP  to  figure  out  who  those  people 
are.  And  able  to  click  on  that  and  having  the  list  of  individuals  who  have  charged  to  that 
[project]  and  maybe  their  total  amount  would  be  wonderful!” 

Impact 

Giving  project  managers  access  to  more  information  about  the  financial  behavior  and  general 
structure  of  their  teams  would  immediately  create  value  and  increase  the  likelihood  of  repeated  use  or 
more  frequent  visits  to  the  product. 

Action  Item 

Investigate  surfacing  information  from  Navy  ERP  about  who  is  charging  to  a  project  and  co¬ 
locating  other  information  about  team  members  inside  the  product. 

ADD  VALUE  THROUGH  CONTENT:  FINANCIAL 
Summary 

Project  managers  expressed  varying  levels  of  access  to  financial  information  for  their  projects. 
Most  project  managers  described  weekly-to-daily  reports  from  their  business  financial  manager.  This 
information  represents  tailored  requests  and  is  a  large  burden  on  the  business  financial  manager  for 
both  consistently  updating  project  managers  and  individually  tailoring  their  output  for  specific 
projects.  This  established  system  has  multiple  facets  and  methodologies,  altering  it  would  be  costly 
and  have  potentially  a  negative  impact. 

Project  managers  did  express  interest  in  an  overview  of  financial  information  about  their  project 
(using  a  line  chart  with  the  expected,  planned,  and  actual  spending  rates.  Spending  rates  are  detailed 
in  Figure  A-2.  These  rates  are  readily  accessible  and  updated  frequently.  The  managers  appreciated 
the  possibility  of  pushing  this  chart  to  management  and  having  it  available  at  all  times.) 
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Project  managers  had  this  to  say: 

“The  distribution,  my  plan  and  my  actuals,  then  I  know  if  I'm  way  off  my  expected  spend 
curve,  but  do  I  have  a  story  because  my  plan  tells  me  that,  for  example,  I  have  a  big  purchase 
coming  up.  [I’d]  rather  see  it  in  a  line  graph  with  those  three  elements.  What  we  roll  up — 
spreadsheet  number,  percentage  expended,  appropriation  plan — three  straight  numbers  also 
raise  a  huge  red  flag  if  I'm  off  my  plan  or  appropriation.  If  I'm  off,  then  I  want  to  drill  down, 
goes  for  the  branch  as  well.  I'd  like  the  line  graph  with  the  ability  to  drill  down  into  the 
different  categories.” 

Impact 

The  financial  burn  rate  of  a  project  is  a  primary  status  indicator  to  both  project  managers  and 
leadership.  Supplying  a  location  where  that  information  is  automatically  and  frequently  updated  may 
take  the  burden  off  the  business  financial  managerss  and  provide  the  project  managers  with  both 
more  situational  awareness  and  the  ability  to  more  easily  brief  the  project  to  others.  However,  if  this 
information  is  not  supplied  automatically  on  the  backend  from  work  the  business  financial  managers 
are  already  doing,  it  will  be  looked  at  as  an  additional  burden  and  will  foster  discontent.  Figure  A-2 
shows  an  overview  of  the  financial  results  for  research  conducted  during  this  session. 


Action  Item 

Investigate  automatically  tracking  financial  information  business  financial  managers  already 
supply  and  display  it  within  the  portal  to  provide  visibility  to  project  managers  about  their  projects’ 
expected,  planned,  and  actual  spending  rates. 
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ADD  VALUE  THROUGH  CONTENT:  RISKS 


Summary 

Most  project  managers  mentioned  a  desire  to  see  project  risks  specified.  They  expressed  concern, 
however,  that  after  the  initial  project  characterization  phase,  the  risk  information  would  become  stale. 
Despite  this  observation,  they  still  felt  it  would  be  excellent  for  roll-up  to  leadership. 

Impact 

Adding  information  about  product  risks  will  add  to  the  expected  value  of  the  portal,  but  there  are 
concerns  about  information  management.  If  maintaining  risk  information  requires  extra  work  from 
the  project  manager  (e.g.,  having  to  maintain  it  in  two  locations),  then  it  loses  value. 

Action  Item 

Investigate  surfacing  risk  information  from  its  current  project  management  tool,  and  other  risk¬ 
tracking  methods  for  potential  interoperability  for  visibility  by  project  managers. 

Figure  A-3  shows  initial  interface  screen  shots  taken  during  the  research  conducted  March  3  4 
2015. 
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Figure  A-3.  Initial  interface  screenshots  for  research  conducted  March  3-4  2015. 
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Figure  A-4  shows  the  pre-interview  form  used  during  the  research  conducted  March  3-4  2015. 


PM  Tools  User  Centered  Design  Participant 

Confirmation 

All  responses  to  this  form  are  anonymized,  and  will  not  be  tied  to  you  in  any  way. 

*  Required 

Please  Enter  Your  Participant  Code  * 

This  is  the  three  digit  number  starting  with  '3'  in  your  confirmation  email. 


How  long  have  you  been  a  project  manager? 

Please  count  from  your  first  assignment  to  your  most  recent. 

O  Less  than  a  year 
O  1-5  years 
O  6-10  years 
O  11 -15  years 
O  16+ years 

On  average  how  many  projects  do  you  typically  manage  at  the  same  time? 

Please  include  any  projects  where  you  are  listed  officially  as  the  project  manger  or  IPT  lead. 

O  1 
O  2-3 
O  4-5 
O  6-10 
O  11  + 

Of  the  following,  which  are  your  top  two  sources  of  information  about  project  managment? 

□  Switchboard 

□  Blog 

□  NSERC  (a.k.a.  Sharepoint) 

□  Wiki 

□  Collegues 

□  Other:  [  | 


Submit 


Never  submit  passwords  through  Google  Forms. 


Powered  by  Thrs  content  is  neither  created  nor  endorsed  by  Google. 

E)  Google  Forms  Report  Abuse  -  Terms  of  Service  -  Additional  Terms 


Figure  A-4.  Pre-interview  form  for  research  conducted  March  3-4  2015. 
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Table  A-2  shows  the  session  script  used  during  the  research  conducted  March  3-4  2015. 


Table  A-2.  Session  script  for  research  conducted  March  3-4  2015. 


Session  Schedule  and  Script 

Time 

Activity 

Pre-Session 

Basic  Questionnaire  Information  (filled  in  for  Moderator) 

Name 

Competency 

Duration  of  PM 

#  of  Projects 

Typical  Info  sources 

First  5  Minutes 

Introduction 

Table  A-3  shows  the  script  interview  table  used  during  the  research  conducted  March  3-4  2015. 


Table  A-3.  Script  interview  table  for  research  conducted  March  3-4  2015. 


Action  and  Time  Line 

Script 

Disclaimer 

Thank  you  for  making  time  in  your  schedule  for  us! 

My  name  is  Christian.  I'm  a  Human  Factors  Engineer  with  Code  536,  and 

I’ve  been  hired  by  leadership  here  at  the  Center  to  help  develop  a  set  of 
User  Requirements  to  guide  and  support  the  development  of  a  suite  of 
tools  supporting  our  project  managers. 

This  is  Sarah;  she’ll  be  taking  notes  throughout  our  session. 

Today,  we  schedule  these  30  minutes  of  you  during  which  I’m  going  to  ask 
you  some  prepared  questions  and  take  you  though  a  few  sketched  out 
designs  of  possibilities  for  a  new  project  management  tool.  1  may  ask  you 
to  “tell  me  more”  or  to  give  me  some  examples.  1  will  do  this  to  ensure  1 
fully  understand  your  answers  and  that  1  can  faithfully  represent  your 
needs  when  1  present  my  findings. 

The  most  important  part  about  your  participation  in  this  interview  is  that 
you  consider  each  question  carefully  and  provide  me  with  as  detailed  a 
response  as  you  can. 

Now,  it’s  important  for  you  to  know  that  you  may  stop  the  interview  at  any 
point.  Similarly,  if  1  ask  you  a  question  that  you’d  rather  not  answer,  simply 
tell  me  and  I’ll  move  on.  It’s  important  that  you  feel  comfortable  throughout 
this  process.  If  you  want  to  stop  at  any  time,  just  let  me  know. 

During  this  interview,  we  will  be  taking  notes  (and  recording)  make  a 
record  of  our  session.  These  notes  and  recordings  are  only  for  our 
reference.  They  are  simply  materials  we  will  use  during  development  of  the 
User  Requirements.  None  of  the  information  you  share  with  us  will  be 
attributed  to  you  and  will  be  anonymized  in  any  supporting  documentation 
or  presentations  we  develop  as  a  result  of  this  session. 

Do  you  have  any  questions  or  concerns  about  this? 

Before  we  begin,  do  you  have  any  questions  about  this  process? 

Are  you  ready  to  begin? 

10  Minutes 

Task  Walkthrough 
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Table  A-3.  Script  interview  table  for  research  conducted  March  3-4  2015  (continued). 


Action  and  Time  Line 

Script 

Introduction 

We  are  going  to  try  what’s  called  “semi-structured  interview,”  which 
means  1  have  a  few  questions  here  I'd  like  to  ask  you  about,  but  we  can 
elaborate  if  we  want. 

Question  1 

So  how  do  you  like  to  manage  your  projects?  Is  there  anything  you  do 
every  day  or  couple  of  days  that  helps  you  stay  on  top  of  it? 

Question  2 

What  project  information  is  your  go-to  status  update? 

Question  2-B 

— >What  software  do  you  generally  use  to  track  that  info? 

Question  3 

What  tvoes  of  proiect  information  do  vou  normallv  qo  to  or 

to  find? 

Question  3-A 

— >IF  INTERNET :  do  you  have  a  couple  of  different  search  terms  you  like 
to  try? 

Question  3-B 

->IF  COLLEAGUES:  Do  you  have  a  go-to  person  you  like  to  ask  project 
management  questions  to? 

Question  3-C 

— ->  IF  YES:  Why  them? 

Question  4 

So  now  that  we’ve  talked  about  it  a  bit,  how  would  you  define  the  part  of 
your  job  that  is  project  management?  Maybe  describe  a  little  of  what  it’s 
like  to  be  you  handling  your  projects? 

5  minutes 

Break  and  Change  Over 

10  Minutes 

Portal  Walkthrough 

Introduction 

When  we  do  this  next  part,  1  want  to  remind  you  we’re  always  testing 
ourselves  at  the  UCD,  never  you.  You  are  giving  us  valuable  insight  into 
the  users  we  want  to  help.  So  be  as  honest  as  you  can,  and  try  not  to  be 
nervous. 

Instructions 

Now,  what  I'm  going  to  show  you  are  digital  sketches  of  what  the  product 
might  look  like.  These  are  early  designs  so  some  might  be  incomplete, 
there  will  be  elements  that  have  fairly  good  detail,  and  there  will  be 
elements  that  will  be  completely  missing.  If  we  run  into  dead  ends,  or  parts 
that  haven't  been  completed,  we  won't  worry  about  it,  we'll  just  move  on. 
That's  super  normal  when  we  test  early  on  like  this. 

1  also  may  ask  you  to  do  things  we  actually  haven't  created  designs  for  yet. 
That’s  fine,  we  want  to  hear  what  you  think  anyway. 

So  when  we  get  started,  remember  to  just  say  what  you're  thinking  out 
loud  and  give  me  your  gut  reaction.  Now  I'm  going  to  pull  up  a  page,  and 
when  1  do,  1  want  you  to  just  tell  me  the  first  thing  you  notice. 

Ready,  set,  [PULL  UP  PAGE] 

ACTION 

Moderator  pulls  up  the  landing  page  (Page  A) 

Question  2 

So  what  did  you  notice  first? 

Question  3 

Looking  at  this  page,  what  do  you  think  you  can  do  with  it? 

QUESTION  4  Show 

Me 

What  would  vou  do  here  to  find  that  info  vou  mentioned  earlier? 

(Info  mentioned  in  semi  structured  interview) 

ACTION 

Moderator  follows  suggestion  and  click  "my  projects"  or  "as  leadership"  or 
talks  through  their  action,  then  clicks  "my  projects"  or  leadership 

ACTION 

My  Projects  or  Leadership  Page  opens 
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Table  A-3.  Script  interview  table  for  research  conducted  March  3-4  2015  (continued). 


Action  and  Time  Line 

Script 

QUESTION  5  (after 
click) 

Is  this  what  you  expected? 

Question  6 

What  caught  your  eye  first? 

QUESTION  7 

What  can  you  do  here? 

QUESTION  8 

Are  you  missing  anything  you  usually  look  for? 

QUESTION  10  Show 
Me 

From  here,  can  you  show  me  how  you'd  find  out  if  project  XYZ  is  close  to 
maxing  out  their  travel  budget? 

ACTION 

Moderator  follows  suaaestion  and  click  "  "  or  talks  throuah  their 

action,  then  clicks " 

ACTION 

Project  Page  Opens 

QUESTION  11  (after 
click) 

Is  this  what  you  expected? 

QUESTION  10 

Tell  me  what  caught  your  eye  first? 

QUESTION  12 

What  can  you  do  here? 
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CONCLUSION 


Initial  user  research  into  the  project  management  portal  revealed  a  strong  need  for  a  clear  value 
proposition  for  the  portal.  Moving  forward,  it  is  critical  to  intuitively  inform  the  users  that  the  portal 
is  focused  on  the  Center’s  project  management  processes,  bringing  value  through  co-location  of  all 
the  tools  and  resources  a  project  manager  will  need  to  accomplish  Center-level  project  management 
process  tasks.  Because  project  managers  primarily  focus  on  accomplishing  their  project  needs 
through  performance,  finance,  and  schedule,  the  burden  falls  on  the  portal  to  deliver  rather  than 
demand  value  and  insert  itself  smoothly  into  the  project  manager’s  existing  activities. 

Project  managers  had  this  to  say  specific  to  their  focus  priority  on  the  Center: 

“The  last  thing  I  am  thinking  about  is  what  the  Center  needs.” 

The  UCD&E’s  recommendations  derive  from  this  initial  user  research  report  and  heuristic 
evaluations  of  the  proposed  product’s  environment,  users,  and  goals.  As  user  research  and  usability 
testing  progress,  recommendations  will  modify  or  increase.  However,  the  UCD&E  team  is  confident 
in  recommending  a  product  plan  that  focuses  on  only  Center  processes  for  project  management, 
while  maintaining  a  consistent  and  professional  user  experience.  Even  though  there  are  opportunities 
for  adding  value  to  the  product,  (see  Findings  Section),  recommendations  related  to  the  primary 
product  goal  must  be  addressed  first. 


USER-CENTERED  DESIGN  AND  ENGINEERING  TEAM 
CONTACT  INFORMATION 

•  Eric  VonColln  eric.voncolln@navy.mil 

•  Sarah  Hunt  sarah.hunt@navy.mil 

•  Christian  Szatkowski  christian.szatkowski@navy.mil 

•  Roni  Higgins  roni.higgins@navy.mil 

•  Jake  Viraldo  jacob.viraldo@navy.mil 
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APPENDIX  B 


USER  RESEARCH  TEST  RESULTS 
APRIL  21-23  2015  AT  SSC  PACIFIC 
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APPENDIX  B 


EXECUTIVE  SUMMARY 

Usability  tests  of  the  low-fidelity  wireframes  conducted  in  April  2015  revealed  that  the  interface 
design  and  the  planned  interaction  models  were  highly  successful.  Users  were  delighted  with  the 
portal’s  look  and  feel  and  felt  optimistic  that  they  could  perform  project  management  tasks  through 
the  interface.  The  burden  remains  to  design  an  interface  that  co-locates  all  tools  and  resources 
required  for  project  management  in  an  easily  understood  and  navigable  format. 

Participants  said  the  following: 

“I  like  the  setup  visually.” 

“If  you  had  everything  [we  needed]  in  there  you’d  have  a  lot  of  happy  project  managers.” 

Future  designs,  recommendations,  and  briefings  by  the  User  Centered  Design  and  Engineering 
(UCD&E)  team  will  address  the  following: 

•  One-stop  shopping:  Decrease  user  workload  and  frustration  while  increasing  buy-in  and 
participation  by  providing  a  single  source  to  complete  desired  tasks 

•  Navigation:  Provide  users  with  a  simple,  clear  navigation  experience 

•  Plain  language:  Use  clear  and  consistent  terminology  to  increase  recognition,  learnability,  and 
user  trust 

•  Support  user  in  task  completion:  Ensure  that  the  system  and  interface  take  the  burden  for 
information  recall  and  task  recognition  off  the  user 

•  Increase  interface  transparency  and  communication:  Tailor  the  interface  to  communicate  value 
specific  to  the  user. 

The  User  Center  Design  and  Engineering  Branch’s  recommendations  are  based  on  carefully 
designed  usability  tests  and  user  research,  as  well  as  on-going  heuristic  evaluations.  As  user  research 
and  testing  progresses,  recommendations  will  be  shared  with  the  development  team  and  others  across 
the  project’s  organization. 

The  UCD&E  team  is  confident  that  its  iterative  design-test-revise-develop  model  is  continuing  to 
produce  many  benefits  as  it  helps  reveal  user  expectations  and  flaws  in  the  product’s  design. 
Additionally,  with  attention  to  UCD&E’s  recommendations,  the  Center’s  user  community  ultimately 
will  be  well  served  with  a  consistent  and  professional  user  experience. 
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PRIMARY  ACTION  ITEMS 


OVERVIEW 

Table  B-l  lists  issues  found  during  the  early  stage  of  portal  research  and  action  items  moving 
forward  to  resolve  them.  Additional  detail  about  each  of  these  items  is  in  the  Findings  Section. 


Table  B-1.  Global  featured  action  items  uncovered  during  initial  user  research. 


Topic 

Action  Item 

Priority 

One-stop  Shopping: 
Decrease  user  workload 
and  frustration  while 
increasing  buy-in  and 
participation  by  providing  a 
single  source  to  complete 
desired  tasks 

The  product  will  include  links  to  the  Work  Acceptance  Tool 
(WAT),  Project  Information  and  Tracking  (PITT),  and  other 

Center  tools,  databases,  asset  repositories,  libraries,  etc.,  and 
as  such  will  satisfy  users’  desire  for  a  “one-stop  shop.”  All 
interactions  with  these  other  tools  will  begin  and  end  with  the 
user  returning  to  the  same  user  experience  as  each  interaction 
is  completed. 

High 

Navigation:  Provide  users 
with  a  simple  clear 
navigation  experience 

Increase  whitespace  and  simplicity  of  the  persistent  navigation 
elements.  Focus  additional  testing  on  difficult  navigational 
tasks,  such  as  returning  to  “my  project”  from  deep  in  “Explore”. 
Maintain  a  clear  visual  hierarchy  between  subordinate  and 
superior  items. 

High 

Plain  Language:  Use  clear 
and  consistent  terminology 
to  increase  recognition, 
learnability,  and  user  trust 

In  a  future  sprint,  the  UCD&E  team  will  conduct  several  new 
studies  to  evaluate  how  well  language,  content,  and  context 
support  the  users’  goals  and  satisfy  the  Center’s  objectives. 
Techniques  of  card  sorting,  card  storming,  A/B  testing,  and 
others  will  be  employed  to  better  understand  how  well  the 
interface  and  interaction  models  support  the  users’  needs  for 
comprehension  and  sense  of  place.  These  tests  will  also 
provide  insight  into  how  well  general  usability  measures  are  met 
as  users  interact  with  the  product. 

High 

Support  User  in  Task 
Completion:  Ensure  that 
the  system  and  interface 
take  the  burden  for 

information  recall  and  task 
recognition  off  the  user 

Ensure  the  product  is  cross-pollinating  information  between  all 
tools,  notifying  user  of  changes,  actions,  and  requirements, 
while  always  providing  clear  information  about  next  steps  or 
pending  deadlines. 

Medium 
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Table  B-1 .  Global  featured  action  items  uncovered  during  initial  user  research  (continued). 


Topic 

Action  Item 

Priority 

Increase  Interface 
Transparency  and 
Communication:  Tailor 

the  interface  to 

communicate  value 

Provide  information  to  the  users  that  is  easy  for  them  to 
understand.  Rely  on  logic  that  follows  users’  models  for  the 
tasks  they  are  performing  in  a  transparent  fashion.  Reveal 
help,  amplifying  information,  and  alternative  workflows 
whenever  necessary. 

Medium 

Orange  =  medium  priority 


Additional  action  items  that  surfaced  during  the  user  research  and  heuristic  reviews  and  which  the 
UCD&E  team  could  address  through  the  design  process  were  incorporated  into  the  baseline  design. 
These  issues  included  inconsistent  use  of  language,  problems  with  the  navigational  structure,  and 
alignment  problems  with  some  of  the  visual  elements. 
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PRODUCT  AND  DESIGN  GOAL 


OVERVIEW 

This  section  contains  product  and  design  goals  used  in  the  portal  evaluation  study.  The  goal  of  this 
initial  product  is  to  provide  a  single  space  for  SSC  Pacific’s  project  managers  to  access  any  Center 
project  management  content  necessary  to  meet  Center  requirements.  Additionally,  users  expect  the 
space  to  provide  customized  project  management  assistance  based  on  the  characteristics  of  the 
projects  a  project  manager  is  handling. 

GOAL  OF  THIS  RESEARCH 

This  research  focused  on  the  initial  structural  and  content  requirements  for  the  product.  To  that 
end,  this  research  assessed  the  product’s  navigation  elements,  its  consistent  use  of  language,  how 
well  it  supports  shared  understanding  of  the  portal’s  underlying  concepts,  and  whether  it  would 
enable  users  to  meet  the  administrative  requirements  with  which  project  managers  must  comply. 
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USERS’  INTERPRETATION  OF  THE  PRODUCT’S  GOAL 

OVERVIEW 

This  section  focuses  on  users’  perception  of  the  portal  and  their  thoughts  of  how  it  fits  into  their 
current  workflow. 

Users  repeatedly  focused  on  tools  and  capabilities  not  in  the  current  scope  of  the  product.  These 
findings  are  documented  as  considerations  for  the  following: 

A.  Adding  value 

B.  Ensuring  increased  user  adoption 

C.  Increasing  leadership’s  ability  to  track  project  data 

The  UCD&E  team  will  continue  to  improve  the  interface  language  and  navigation  to  properly 
communicate  the  capabilities,  goals,  and  value  of  the  interface  as  currently  envisioned.  If  project 
manager’s  expectations  for  project  management  and  project  process  support  are  not  realized, 
however,  project  managers  will  lose  confidence  in  the  overall  product  suite  and  the  adoption  rate  will 
be  low. 

INCREASED  MANAGEMENT  CAPABILITIES:  USERS  ARE  HIGHLY  INTERESTED  IN  A  TOOL 
TO  HELP  THEM  MANAGE  PROJECTS  BETTER 

At  the  heart  of  a  project  manager’s  work  is  management.  Tools  and  spaces  that  allow  the  manager 
to  more  effectively  manage  their  projects  are  of  primary  concern  in  any  context.  Increased 
management  capabilities  are  of  great  importance  to  users.  Increased  focus  on  developing  a  portal 
with  a  fully  functioning  project  dashboard  workspace  is  key  to  giving  project  managers  a  useful  tool. 

Project  managers  had  this  to  say: 

“I  need  a  supporting  product,  at  the  product  manager  level,  more  than  ...  a  roll-up  tool  to  be 
monitored  and  measured  from  above.” 

FACILITATE  COLLABORATION  AND  SHARING:  USERS  DESIRE  THE  ABILITY  TO  SHARE 
RESOURCES  WITH  OTHER  USERS 

Another  common  thread  in  the  study  was  the  idea  of  open  communication  and  collaboration 
between  project  managers  across  the  Center.  Collaboration  was  valued  highly  among  project 
managers  as  a  desired  feature.  Project  managers  want  to  view  and  interact  with  their  peers  across  the 
Center  using  digital  open  houses,  project  videos,  published  quad  charts,  personnel  organizational 
charts,  and  shared  templates. 

Project  managers  had  this  to  say: 

“High-level  information  should  be  accessible  to  everyone  at  the  Center  ...  to  enable 
collaboration.” 

Providing  project  managers  with  easy,  reliable  access  to  channels  is  highly  valued  to  project 
managers.  They  want  to  easily  publish  project  documents  and  artifacts  supports.  They  also  want  tools 
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in  place  that  support  Center-wide  cooperation  and  collaboration  efforts.  They  want  a  space  that 
allows  them  to  share  resources  that  they  believe  will  improve  their  overall  productivity  and  increase 
their  ability  to  share  resources. 

MINIMIZE  USER  TASKS:  LEVERAGE  PROJECT  MANGERS’  SCARCE  RESOURCES  FOR  A 
CLEAR  RETURN  ON  INVESTMENT 

A  project  manager’s  primary  responsibility  is  first  to  the  sponsor,  the  project,  and  the  team.  Unless 
a  return  on  investment  is  evident  concerning  those  priorities,  any  work  product  that  the  manager 
believes  is  not  delivering  a  return,  is  likely  to  be  completed  half-heartedly,  or  not  at  all. 

Project  managers  had  this  to  say: 

“The  ROI  for  the  time  I  put  into  it  is  zero,  it’s  negative,  actually.  I  record,  I  do  what  they’re 
looking  for,  but  I  don’t  feel  that  the  completion  of  that  adds  benefit  to  how  I’m  running  the 
project.” 

To  combat  this,  return  on  investment  should  either  be  increased,  or  better  communicated  to  the 
performer.  Alternatively,  if  possible,  the  task  should  be  eliminated.  Only  in  extreme  cases  should  it 
be  mandated;  unless  it  is  deemed  worthy  of  users’  attention,  it  will  likely  be  regarded  as  worthless  or 
never  completed  at  all. 
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USERS 


Participant  survey  questions  are  shown  in  Table  B-2.  This  round  of  research  included  six 
50-minute  participant  interviews.  The  study  participants  were  characterized  according  to  the 
following  dimensions: 

•  Gender:  both  male  and  female 

•  Experience:  Less  than  a  year  up  to  16  plus  years 

•  Number  of  Concurrent  Projects:  1-3 

•  Education:  Bachelors-Doctorate 

•  Government  service:  3-32  years 

•  Codes:  53,  54,  and  56 


Table  B-2.  Participant  survey  questions  concerning  project  management. 


On  average,  how  many 

How  often  do  you  delegate  project 

Have  you  ever  completed 

projects  do  you  typically 

management  tasks  like  filling  out 

a  project  management 

manage  at  the  same  time? 

the  WAT  or  PIT  to  another  person? 

plan  (PMP)? 

2-3 

'A  of  the  time 

Yes 

1 

Never 

Yes 

1 

Never 

No 

2-3 

Never 

Yes 

1 

Never 

Yes 

2-3 

V4  of  the  time 

Yes 
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METHODOLOGIES 


OVERVIEW 

This  section  contains  three  methods  that  were  used  for  the  portal  study.  This  study  contains  a 
second  series  of  usability  tests  that  included  a  pre-session  data  collection  that  was  conducted  both 
online  and  through  email.  Each  test  session  included  a  semi-structured  interview  followed  by  a 
cognitive  walkthrough  of  several  updated  wireframes.  Each  test  took  approximately  50  minutes. 
Researchers  used  the  results  to  evaluate  how  well  wireframe  changes  supported  the  needs  of  the 
targeted  users. 

Step  1:  Pre-Interview  Data  Collection 

Prior  to  the  user  interviews,  participants  answered  some  basic  questions  concerning  the  amount  of 
time  they’ve  been  a  project  manager,  the  number  of  projects  they  typically  manage,  the  frequency 
they  delegate  project  management  tasks,  and  whether  they  had  ever  completed  a  project  management 
plan.  See  Figure  B-l  for  an  example  of  the  complete  participant  survey. 

Step  2:  Semi-Structured  Interview 

Each  test  session  began  with  a  semi-structured  interview.  These  interviews  included  specific 
scripted  questions  with  the  addition  of  optional  probing  questions  (e.g.,  “Tell  me  more...”)  that 
allowed  participants  to  elaborate  on  their  answers.  These  interviews  were  designed  to  capture  basic 
information  about  the  participants’  project  management  preferences,  basic  knowledge  of  Center 
processes,  project  management  priorities,  project  management  tools,  and  general  methodologies.  See 
Figure  B-2  for  an  example  of  the  pre-interview  protocol  used.  Figure  B-3Error!  Reference  source 
not  found,  shows  the  session  script  used  during  the  research  conducted  April  21-23  2015. 

Step  3:  Initial  Interface  Guided  Walkthrough 

This  part  of  the  research  was  designed  to  elicit  feedback  on  how  well  the  updated  wireframes  met 
the  participants’  mental  models  and  usage  expectations.  Additional  goals  included  evaluating  the 
navigational  elements,  labels,  and  goal-based  content  presented  in  the  wireframes  and  determining 
the  overall  usefulness  of  the  interface.  This  evaluation  was  conducted  in  a  semi-structured  manner, 
giving  the  moderator  a  chance  to  explore  a  participant’s  response  off-script  to  gain  a  deeper 
understanding  of  the  participant’s  remarks. 
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_ Participant  Code 

How  long  have  you  been  a  project  manager? 

Please  count  from  your  first  assignment  to  your  most  recent 

□  Less  than  a  year 

□  1-5  years 

□  6-10  years 

□  11 -15  years 

□  16+ years 

On  average,  how  many  projects  do  you  typically  manage  at  the  same  time? 

Please  include  any  projects  where  you  are  listed  as  the  project  manager  or  IPT  lead 

□  1 

□  2-3 

□  4-5 

□  6-10 
□  11  + 

How  often  do  you  delegate  project  management  tasks  like  filling  out  the  WAT  or  PIT  to 
another  person? 

□  Always 

□  %  of  the  Time 

□  Half  the  Time 

□  V4  of  the  Time 

□  Never 

Have  you  ever  completed  a  project  management  plan  (PMP)? 

□  Yes 

□  No 

□  I  don’t  know  what  that  is 

Figure  B-1 .  Participant  survey. 
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Thank  you  for  making  time  in  your  schedule  for  us! 

My  name  is _ .  I’m  a  Human  Factors  Engineer  in  the  UCD&E  group  and  we’re  helping  design 

a  suite  of  tools  to  help  support  our  project  managers. 

This  is _ s/he’ll  be  taking  notes  throughout  our  session. 

Today  we’d  like  to  learn  more  about  how  well  these  preliminary  designs  can  support  you  as  a  PM  or 
IPT  lead.  I’ll  start  by  asking  you  some  prepared  questions  and  taking  you  though  a  few  low-fidelity 
designs.  During  this  time,  I’m  going  to  encourage  you  to  think  out  loud,  and  verbally  walk  through  your 
reactions  and  decision  making.  I  may  ask  you  to  “tell  me  more”  or  to  give  me  some  examples.  I  will  do 
this  to  ensure  I  fully  understand  your  answers  and  to  help  me  faithfully  represent  your  needs  when  I 
present  my  findings. 

The  most  important  part  about  your  participation  in  this  interview  is  that  you  consider  each  question 
carefully  and  provide  me  with  as  detailed  a  response  as  you  can. 

Now,  it’s  important  for  you  to  know  that  you  may  stop  the  interview  at  any  point.  Similarly,  if  I  ask  you  a 
question  that  you’d  rather  not  answer,  simply  tell  me  and  I’ll  move  on.  It’s  important  that  you  feel 
comfortable  throughout  this  process.  If  you  want  to  stop  at  any  time,  just  let  me  know. 

During  this  interview,  we  will  be  taking  notes  (and  recording)  in  order  to  make  a  record  of  our  session. 
These  notes  and  recordings  are  only  for  our  reference.  They  are  simply  materials  we  will  use  as  we 
revise  our  design  and  update  User  Requirements.  None  of  the  information  you  share  with  us  will  be 
attributed  to  you  and  will  be  anonymized  in  any  supporting  documentation  or  presentations  we  develop 
as  a  result  of  this  session. 

Do  you  have  any  questions  or  concerns  about  this? 

Before  we  begin,  do  you  have  any  questions  about  this  process? 

Are  you  ready  to  begin? 

Figure  B-2.  Pre-interview  protocol. 
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Figure  B-3  shows  a  general  semi-structured  interview  session  script 


General  Semi-Structured  Interview: 

To  begin,  we  are  going  to  try  what’s  called  “semi-structured  interview”,  which  means  I  have  a  few 
questions  here  I'd  like  to  ask  you  about,  but  we  can  elaborate  if  we  want. 

Question  1 :  So  how  do  you  like  to  manage  your  projects?  Is  there  anything  you  do  every  day  or  couple 
of  days  to  help  you  stay  on  top  of  it? 

Question  2:  How  well  do  you  know  the  project  processes  the  Center  expects  you  to  complete? 

'— >  Where  do  you  normally  go  to  learn  about  these  processes? 

Question  3:  What  tools  do  you  generally  use  to  manage  the  Center  processes  for  your  project? 

'— >  Where  do  you  normally  go  to  access  these  tools? 

'— >  What  is  your  opinion  of  these  tools? 

'— >  What  would  make  them  easier  to  use?  (Prompt,  if  necessary) 

Question  4:  So  now  that  we’ve  talked  about  it  a  bit,  tell  me  again:  how  would  you  define  the  part  of 
your  job  that  is  project  management?  Maybe  describe  a  little  of  what  it’s  like  to  be  you  handling  your 
projects? 

*****5  minute  break***** 

Walkthrough 

For  this  next  part,  we  will  show  you  one  simple  concept  rendering  of  a  landing  page  for  a  project 
manager  Tool;  we  are  interested  in  understanding  how  you  would  use  it.  Now,  what  I’m  going  to  show 
you  are  digital  sketches  of  what  the  product  might  look  like.  These  are  early  designs  so  some  might  be 
incomplete,  there  will  be  elements  that  have  fairly  good  detail,  and  there  will  be  elements  that  will  be 
completely  missing. 

So  when  we  get  started,  remember  to  just  keep  saying  what  you're  thinking  out  loud  and  give  me  your 
gut  reaction.  Just  tell  me  the  first  thing  you  notice. 

Display  Landing  Page 

Ready,  set,  [PULL  UP  PAGE] 

Figure  B-3.  General  semi-structured  interview  session  script. 
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Figures  B-4  through  B-9  show  the  customer  interface  feedback  questionnaire  used  during  the 
research  conducted  April  21-23  2015. 


Ready,  set.  (PULL  UP  PAGE] 

So  what  did  you  notice  first? 

Looking  at  this  page,  what  do  you  think  you  can  do  with  it? 

Where  do  you  wantto  gofirst?  Whatshouldhappenifyouclickthere? 
Where  do  you  wantto  go  next?  What  should  happen  if  you  click  there? 
Etc. 

What  is  missing  fromthis  page? 

Figure  B-4.  Customer  interface  feedback  questionnaire  (page  1). 
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Where  do  you  think  you  would  go  first  to _ ? 

Choose  at  random  —  (Add  a  new  team  member,  add  a  new  administrator,  request  removal 
from  a  project) 

♦♦♦after  click*** 

Is  this  what  you  expected? 

What  caught  your  eye  first? 

What  can  you  do  here? 

Are  you  missing  anything  you  usually  look  for? 

Figure  B-5.  Customer  interface  feedback  questionnaire  (page  2). 


B-16 


Where  do  you  think  you  would  go  first  to _ ? 

Choose  at  random  —  (Add  a  new  team  member,  add  a  new  administrator,  request  removal 
from  a  project) 

***after  click*** 

Is  this  what  you  expected? 

What  caught  your  eye  first? 

What  can  you  do  here? 

Are  you  missing  anything  you  usually  look  for? 

Figure  B-6.  Customer  interface  feedback  questionnaire  (page  3). 
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Figure  B-7.  (a)  Landing  page;  (b)  Explore-browse-selection  tool  (i.e.,  “sandbox”). 
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Figure  B-8.  (a)  Project  administration;  (b)  Add  a  new  team  member  popup. 


Figure  B-9.  (a)  Add  a  new  administrator  popup;  (b)  request  removal  format. 
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FINDINGS 


OVERVIEW 


This  section  contains  findings  found  during  all  parts  of  the  study  for  this  report.  In  this  section, 
findings  are  briefly  described  with  a  summary  of  the  findings.  Comments  from  portal  users  specific 
to  the  issue  are  shown,  and  an  action  item  is  given  of  plans  to  correct  the  issue. 

The  section  describes  the  users’  overarching  expectations  for  a  future  tool  or  project  management 
space  as  well  as  disappointments  with  current  tools  and  Center-level  project  management  practices. 
During  this  study,  the  UCD&E  team  identified  many  separate  but  related  needs  that  turned  up 
repeatedly  in  the  participants’  user  stories,  both  in  open  interviews  and  during  the  task-oriented 
walkthroughs.  Results  are  grouped  into  nine  categories. 

SUMMARY  OF  FINDINGS 

The  chart  shown  in  Figure  B-10  details  eight  generalized  categories  of  findings  revealed  during  the 
April  2015  user  studies.  Included  in  this  chart  is  a  segment  called  “Extra  Features,”  which  represents 
user  feedback  on  desired  tool  capabilities  such  as  directory  look  up,  document  repository,  Google- 
styled  search,  and  animated  webpage  tours. 


One-Stop  Shopping,  13  SharinS  ^sources, 16 


Extra  Features,  34 


Increased  Management 
"  Capabilities,  21 


Navigation,  10 


Minimize  User  Tasks,  20 


Figure  B-10.  Number  of  responses  per  topic. 
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ONE-STOP  SHOPPING 


The  goal  of  one-stop  shopping  was  to  decrease  user  workload  and  frustration  while  increasing  buy- 
in  and  participation  by  providing  a  single  source  to  complete  desired  tasks. 

Summary 

Users  want  “one  place”  to  go  to  manage  projects  and  the  users’associated  project  management 
requirements. 

Users  uniformly  and  repeatedly  expressed  their  desire  to  have  a  single  point  of  entry  to  access 
tools  and  see  materials  to  help  them  manage  their  projects  and  meet  Center  expectations.  They 
expressed  concern  about  how  hard  it  is  to  navigate  between  the  different  tools  that  currently  exist. 
They  also  want  help  and  assurance  that  they  are  completing  all  of  the  necessary  steps  correctly. 

Participants  said  the  following: 

“...I  just  try  to  stay  off  all  those  dink  lists.” 

Users  currently  employ  a  mixture  of  tools  and  resources  to  execute  their  project  management  tasks, 
including  the  switchboard,  email  links,  and  browser  bookmarks  to  find  their  way  to  existing  Center 
tools  such  as  the  WAT  and  the  PITT.  A  surprising  finding  is  that  even  highly  experienced  users 
claim  that  they  don’t  feel  like  they  have  a  thorough  knowledge  of  all  of  the  capabilities  and  tools  that 
are  available  to  support  them.  This  means  that  users  rely  on  email  reminders  and  data  calls,  which  put 
them  in  a  reactive  stance,  rather  than  proactively  managing  processes  within  a  single,  multipurpose 
tool. 

Participants  said  the  following: 

“That’s  one  thing  I  think  would  really  benefit  everybody,  is  if  you  just  had  one  place  you 
went  to  you  could  find  all  the  tools.” 

Action  Item 

The  portal  needs  to  include  links  to  the  WAT,  PITT,  and  other  Center  tools,  databases,  asset 
repositories,  libraries,  etc.  There  needs  to  be  emphasis  to  satisfy  users’  desires  for  a  “one-stop  shop” 
workspace.  The  user  should  not  have  to  go  to  one  place  for  one  needed  answer  and  elsewhere  for 
another  and  then  have  to  combine  the  solutions  in  the  portal.  All  interactions  with  these  other  tools 
will  begin  and  end  with  the  user  returning  to  the  same  user  experience  as  each  interaction  is 
completed. 

NAVIGATION:  PROVIDE  USERS  WITH  A  SIMPLE  CLEAR  NAVIGATION  EXPERIENCE 
Summary 

Simple  and  clear  navigation  is  critical  to  user  satisfaction,  trust,  and  task  completion.  If  a  user 
cannot  successfully  navigate  around  an  interface,  the  user  cannot  move  forward  and  is  likely  to 
abandon  a  task.  During  this  study  the  goal-based  navigational  language  implemented  for  the 
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“Manage”  and  “Explore”  sections  was  successful;  users  could  easily  locate  their  project  home  page, 
and  navigate  quickly  to  the  interface  from  which  they  can  ask  to  join  a  project. 

Participants  said  the  following: 

“Manage,  oh,  OK  this  is  what  we  need  . . .  and  ‘Request  to  Join  a  Project,’  that  is  exactly  what 
we  do  in  JIRA.  I  can  use  that.” 

Additional  navigational  elements  are  now  under  redesign,  with  focus  on  the  reduction  of  the 
header  when  the  user  has  navigated  past  the  landing  page,  inside  the  interface.  Results  showed  that 
the  persistent  navigation,  as  currently  designed,  was  overly  salient,  distracting  the  user  from  their 
primary  task.  Figure  B-l  1  shows  the  portal  Explore  Center  Project  Management  Best  Practices 
screenshot. 


Figure  B-1 1 .  Explore  Center  Project  Management  Best  Practices  screenshot. 

Action  item 

1 .  Increase  whitespace  and  simplicity  of  the  persistent  navigation  elements. 

2.  Focus  additional  testing  on  difficult  navigational  tasks,  such  as  returning  to  “My  Project” 
from  deep  in  “Explore”. 

3.  Maintain  a  clear  visual  hierarchy  between  subordinate  and  superior  items. 

PLAIN  LANGUAGE:  USE  CLEAR  AND  CONSISTENT  TERMINOLOGY  TO  INCREASE 
RECOGNITION,  LEARNABILITY,  AND  USER  TRUST 

Summary 

To  successfully  use  any  tool  or  information  space,  users  need  the  labels  and  content  to  effectively 
use  descriptive  language  that  has  a  clear  purpose,  is  internally  consistent,  and  aligns  with  their 
definitions  of  terms.  Since  the  project  portal  will  be  used  by  a  variety  of  users  with  various  ranges  of 
project  management  and  Center  experience,  the  UCD&E  team  is  tailoring  it  to  be  self-educating  and 
walk-up.  The  interface’s  language  cannot  make  assumptions  about  the  project  management 
experience  of  the  user  and  must  convert  specialized  terminology  into  common  terms. 
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When  menus,  labels,  banners,  descriptions,  and  temporal  context  are  simple,  consistent,  and 
resonate  with  users’  expectations,  all  users  experience  an  improved  overall  user  experience.  Effective 
use  of  language,  content,  and  context  also  increases  users’  confidence  in  comfortably  navigating  and 
interacting  with  the  tools  and  information  spaces  they  use. 

Participants  said  the  following: 

“I’m  not  sure  of  the  use  of  the  term  ‘project’  here  when  I  see  it.” 

Example  “Communicate” 

The  UCD&E  team  is  focusing  on  driving  a  plain,  goal-based  language  model  throughout  the 
interface  for  proper  user  learning  and  recognition.  However,  one  consistent  problem  faced  during 
tests  was  reconciling  differences  in  users’  mental  models  for  project  management  tasks.  Users  liked 
having  a  stable  link  to  the  Project  Implementation  Tool  on  the  landing  page,  and  the  idea  of  a 
collection  of  tools  all  based  out  of  the  same  location.  However,  the  word  “Communicate”  did  not 
accurately  reconcile  the  users’  mental  model  with  the  type  of  actions  and  goals  they  would  be 
accomplishing  with  these  tools.  Figure  B-12  shows  an  example  of  communication  tools  used. 


The  UCD&E  team  will  take  steps  to  find  the  appropriate  language  to  represent  users’  mental 
models  on  the  page  while  also  retaining  the  value  they  found  in  the  functionality  of  this  feature. 

Action  Item 

In  a  future,  the  UCD&E  team  will  conduct  several  new  studies  to  evaluate  how  well  language, 
content,  and  context  support  the  users’  goals  and  satisfy  the  Center’s  objectives.  Techniques  of  card 
sorting,  card  storming,  A/B  testing,  and  other  methods  will  be  employed  to  better  understand  how 
well  the  interface  and  interaction  models  support  the  users’  needs  for  comprehension  and  sense  of 
place.  These  tests  will  also  provide  insight  into  how  well  general  usability  measures  are  met  as  users 
interact  with  the  product. 
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In  addition  to  ensuring  that  plain  language  and  universal  contexts  are  employed,  the  UCD&E  team 
will  look  into  drafting  an  interactive  glossary  of  terms  and  definitions  that  is  built  out  over  time  for 
reference  by  the  users. 

SUPPORT  USER  IN  TASK  COMPLETION:  ENSURE  THAT  THE  SYSTEM  AND  THE  INTERFACE 
TAKE  THE  BURDEN  FOR  INFORMATION  RECALL  AND  TASK  RECOGNITION  OFF  THE  USER 

Summary 

Users  are  avoiding  completing  Center  process  tasks  due  to  problems  such  as  difficulty  in 
understanding  Center  requirements,  unfunded  mandates,  unclear  or  conflicting  directions,  and 
defective  or  problematic  technical  system.  Users  wait  for  a  mandate  through  another  individual  (like 
a  branch  head)  to  notify  them  of  the  next  required  action. 

Participants  said  the  following: 

“Honestly,  I  wait  for  somebody  to  ask  for  something.  I  have  a  basic  understanding  of  my 
requirements  . . .  the  project  management  plan,  the  completion  of  the  PITT,  the 
characterization  tool  ...  if  there  are  other  requirements  beyond  that,  I  don’t  go  looking  for 
them.” 

Given  their  limited  resources  and  the  numerous  constraints  on  those  resources,  project  managers 
will  do  as  few  difficult  and  non-value-added  activities  as  possible.  They  may  entirely  give  up  on  an 
activity  when  it  includes  what  they  believe  are  insurmountable  obstacles,  in  favor  of  more  productive 
endeavors. 

Action  Item 

Ensure  the  product  is  cross-pollinating  information  between  all  tools,  notifying  users  of  changes, 
actions,  and  requirements.  The  portal  needs  to  provide  clear  information  about  next  steps  or  pending 
deadlines. 

INCREASE  INTERFACE  TRANSPARENCY  AND  COMMUNICATION:  TAILOR  THE  INTERFACE 
TO  COMMUNICATE  VALUE  SPECIFIC  TO  THE  USER 

Summary 

Center  processes  that  apply  differently  to  different  project  managers  and  to  different  projects  need 
to  be  communicated  through  a  simple  transparent  tool. 

Research  suggests  that  user  adoption  may  increase  if  the  results  of  the  Process  Tailoring  Tool 
(PTT)  provided  more  transparency  into  the  results  and  how  they  were  determined. 

Project  managers  had  this  to  say: 

“I  don’t  feel  like  I  have  a  personally  tailored  ...  workbook  that  tells  me  what  to  do.” 

Until  users  feel  that  the  available  project  management  toolset  realistically  reflects  their  project’s 
unique  situation,  they  will  not  completely  trust  the  results  of  a  “one  size  fits  all”  approach. 
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Demonstrating  the  trustworthiness  of  the  tailoring  tool,  and  ensuring  that  it  is  less  painful  for  users  to 
use  will  also  increase  buy-in.  This,  in  turn,  will  increase  the  adoption  rate  and  ultimately  have  a 
positive  effect  on  users’  compliance  using  this  and  other  tools. 

Participants  said  the  following: 

“I’ve  found  it  challenging  to  be  clear  on  exactly  what  it  is  I  need  to  do  ...  Show  me  the  list  of 
what  I  really,  no  kidding,  truly  need  to  do.” 

Action  Item 

Provide  information  that  is  easy  for  users  to  understand.  Rely  on  logic  that  follows  users’  models 
for  the  tasks  they  are  performing  and  does  so  in  a  transparent  fashion.  Reveal  help,  amplifying 
information,  and  alternative  whenever  necessary. 
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CONCLUSION 


Overall,  the  April  usability  tests  of  the  low-fidelity  wireframes  revealed  that  the  interface  design  and 
the  planned  interaction  models  were  highly  successful.  Users  were  delighted  with  the  look  and  feel  and 
felt  optimistic  that  they  could  perform  their  project  management  tasks  through  the  interface. 

Although  the  overall  design  tested  very  well,  flaws  associated  with  the  use  of  plain  language  and 
clear  context  were  observed  in  the  “Communication”  element  of  the  wireframe.  Additionally, 
continued  user  research  in  this  second  round  of  usability  tests  highlighted  again  the  need  for  a  strong 
and  clear  value  proposition  for  the  product.  As  noted  in  a  previous  report  (Final  Report  of  Initial  UCD 
Portal  User  Research  Results,  published  on  March  23,  2015),  providing  users  only  with  a  product 
focused  solely  on  the  Center’s  project  management  processes  brings  little  to  no  value  to  targeted  users 
within  the  project  manager  and  integrated  product  team  lead  population  segments. 

In  order  to  bring  value  to  these  critical  users,  co-locating  all  of  the  tools  and  resources  a  project 
manager  will  need  to  accomplish  his  Center-mandated  project  management  process  tasks  is  essential. 
Because  project  managers  primarily  focus  on  accomplishing  their  project  needs  for  performance, 
finance,  and  schedule,  and  because  they  must  attend  to  the  priorities  of  their  sponsors  first  and 
foremost,  the  burden  falls  on  the  project  portal  to  deliver  value,  not  to  demand  compliance.  The 
portal’s  value  can  only  occur  when  it  inserts  itself  smoothly  into  the  project  manager’s  existing 
activities,  rather  than  existing  as  another  onerous  task  that  must  be  endured. 

Project  managers  had  this  to  say: 

“There  was  a  push  to  get  your  stuff  into  the  PITT,  and  I  said,  ‘okay,  I’ve  seen  so  many  emails 
on  it,  I’ll  try.’  so  I  went  in,  and  nothing  worked.  I  tried  to  upload,  I  tried  to  download  . . .  this 
was  after  I  finally  found  the  tool.  Anyway,  I  finally  found  it,  did  a  bunch  of  stuff;  nothing 
worked. ..  A  couple  weeks  later,  they  fixed  that,  I  uploaded  a  couple  documents  ...  I  didn’t  see 
what  good  that  was  doing...  It’s  like  an  extra  activity  that  seems  to  have  no  particular 
purpose.” 

The  User-Centered  Design  and  Engineering  Branch’s  recommendations  are  based  on  carefully 
designed  usability  tests  and  user  research,  as  well  as  on-going  heuristic  evaluations.  As  user  research 
and  usability  testing  progress,  recommendations  will  be  shared  with  the  development  team  and  others 
across  the  project’s  organization. 
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The  UCD&E  team  is  confident  that  its  iterative  design-test-revise-develop  model  is  continuing  to 
produce  many  benefits  as  it  helps  reveal  user  expectations  and  flaws  in  webpage  designs.  Additionally, 
with  attention  to  UCD&E’s  recommendations,  the  Center’s  user  community  ultimately  will  be  well 
served  with  a  consistent  and  professional  user  experience. 


USER-CENTERED  DESIGN  AND  ENGINEERING  TEAM 
CONTACT  INFORMATION 

•  Deborah  Gill-Hesselgrave  deborah.d.gill@navy.mil 

•  Roni  Higgins  roni.higgins@navy.mil 

•  Sarah  Hunt  sarah.hunt@navy.mil 

•  Christian  Szatkowski  christian.szatkowski@navy.mil 

•  Jake  Viraldo  jacob.viraldo@navy.mil 

•  Eric  VonColln  eric.voncolln@navy.mil 
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APPENDIX  C 


EXECUTIVE  SUMMARY 

Usability  tests  of  early  portal  iterations  conducted  in  May  2015  continued  to  show  successes, 
especially  in  navigation,  interface  design,  and  user  task  performance.  Users  expressed  delight  with 
the  look  and  feel  of  the  portal  and  demonstrated  that  they  could  perform  basic  initial  project 
management  tasks  through  the  interface  with  few  issues.  Going  forward,  the  User-Centered  Design 
and  Engineering  (UCD&E)  Branch  will  continue  to  refine  these  designs,  leveraging  the  lessons 
learned  from  this  round  of  testing.  The  ultimate  goal  is  to  improve  a  portal  interface  design  that 
co-locates  tools  and  resources  required  for  project  management  in  an  easily  understood  and  navigable 
format. 

Participant  327,  while  downloading  a  project  roster  commented: 

“This  is  fun!” 

Future  designs,  recommendations,  and  briefings  by  the  User-Centered  Design  &  Engineering 
Team  will  address  the  following: 

•  Removing,  de-emphasizing,  or  reducing  distractions  caused  by  the  project  management  body 
of  knowledge  (PMBOK)  activities  and  process  framework  overview 

•  Improving  placement  and  timeliness  of  feedback  and  feed-forward  mechanisms 

•  Using  clear  and  consistent  terminology  to  increase  recognition,  learnability,  and  user  trust 

The  UCD&E  Branch’s  recommendations  are  based  on  carefully  designed  usability  tests  and  user 
research,  as  well  as  ongoing  heuristic  evaluations.  As  user  research  and  usability  testing  progress, 
recommendations  will  be  shared  with  the  development  team  and  others  across  the  project’s 
organization. 

Participant  523,  while  using  the  portal  to  create  a  project  commented: 

“Pretty  easy!” 

The  UCD&E  team  is  confident  that  its  iterative  design-test-revise-develop  model  is  continuing  to 
produce  many  benefits  as  it  helps  reveal  user  expectations  and  flaws  in  the  product’s  design.  With 
attention  to  UCD&E’s  recommendations,  the  Center’s  project  portal  user  community  will  be  well 
served  with  a  consistent  and  professional  user  experience. 
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PRIMARY  ACTION  ITEMS 


OVERVIEW 

This  section  contains  a  listing  of  action  items  found  during  the  early  stage  of  portal  research.  Table 
C-l  details  global  or  feature  action  items  uncovered  during  the  initial  user  research.  For  additional 
detail  about  each  of  these  items,  see  the  Findings  Section. 


Table  C-1 .  Global  featured  action  items  uncovered  during  initial  user  research. 


Topic 

Action  Item 

Priority 

Remove,  de-emphasize, 
or  reduce  distractions 
caused  by  the  project 
management  body  of 
knowledge  (PMBOK) 
activities  and  process 
framework  overview 

Because  of  the  distracting  nature  of  the  process  framework  and 
accompanying  PMBOK  activity  map,  the  UCD&E  team 
suggests  removing  them  from  the  portal  entirely.  If  the  product 
owners  insist  on  retaining  one  or  the  other,  we  suggest 
retaining  the  process  framework  while  de-emphasizing  the 
shapes  and  colors  in  a  way  that  shows  users  that  it  is  a 
reference  diagram,  and  not  an  active  page  navigation  element. 

High 

Improve  placement  and 
timeliness  of  feedback  and 

feed-forward  mechanisms 

Virtually  every  participant  expressed  or  otherwise  indicated  the 
desire  for  more  comprehensive,  timely  feedback.  Equally 
important  are  sufficient  feed-forward  mechanisms,  put  in  place 
to  help  manage  users’  expectations  and  improve  navigability. 
Hesitation,  insecurity,  and  excessive  scanning  for  cues  are  all 
symptoms  of  this  condition.  More  feedback  is  needed, 
especially  upon  completion  of  tasks  such  as  project  creation  or 
responding  to  join  requests. 

High 

Use  clear  and  consistent 
terminology  to  increase 
recognition,  learnability, 
and  user  trust 

Continued  improvement  in  this  area  is  a  key  element  of  driving 
good  design.  Words  such  as  “administrator”,  “activities”,  and 
“personnel”  need  to  be  examined  and  retested,  as  necessary. 

Medium 

Additional  action  items  that  surfaced  during  the  user  research  and  heuristic  reviews,  and  which  the 
UCD&E  team  could  address  through  the  design  process,  were  incorporated  into  the  baseline  design. 
These  issues  included  inconsistent  use  of  language,  problems  with  the  navigational  structure,  and 
alignment  problems  with  some  of  the  visual  elements. 
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PRODUCT  AND  DESIGN  GOAL 


OVERVIEW 

This  section  contains  product  and  design  goals  that  were  used  in  the  portal  evaluation  study.  The 
goal  of  this  product  is  to  provide  a  single  space  for  Space  and  Naval  Warfare  Systems  Center 
Pacific’s  (SSC  Pacific)  project  managers  to  access  any  Center  project  management  content  necessary 
to  meet  Center  requirements.  Additionally,  users  expect  the  space  to  provide  customized  project 
management  assistance  based  on  the  characteristics  of  the  projects  a  project  manager  is  handling. 

This  research  focused  on  the  tasks  associated  with  creating  a  joining,  and  managing  personnel  in 
the  project  portal.  It  used  task-based  analysis,  as  well  as  a  think-aloud  protocol  to  examine  the 
facility  with  which  users  performed  a  variety  of  basic,  fundamental  tasks  that  they  would  require  of 
any  new  project.  Additionally,  research  conducted  A/B  tests  designed  to  examine  users’  preferences 
and  abilities  to  properly  differentiate  between  various  methods  of  displaying  and  navigating  certain 
design  elements  within  the  portal.  Elements  of  navigation,  language  usage,  mental  mapping,  and 
interportal  communication  were  tested  and  recorded  to  capture  successes  and  failures  of  the  current 
portal  design. 

The  tasks  included  as  part  of  the  testing  protocol  were  designed  to  elicit  feedback  on  how  well  the 
project  portal  met  the  participants’  mental  models  and  usage  expectations.  Additional  goals  included 
evaluating  the  navigational  elements,  labels,  and  goal-based  content  presented  in  the  wireframes,  and 
determining  the  overall  usefulness  of  the  interface. 

To  the  extent  that  users  focused  on  tools  or  capabilities  not  in  the  current  scope  of  the  product,  the 
UCD&E  team  documented  these  considerations  for  future  use  in  adding  value  and  facilitating  user 
adoption  of  the  portal.  The  UCD&E  team  will  continue  to  improve  the  interface  language  and 
navigation  to  properly  communicate  the  capabilities,  goals,  and  value  of  the  interface  as  currently 
envisioned.  If  project  manager’s  expectations  for  project  management  and  project  process  support  are 
not  realized,  then  project  managers  will  likely  lose  confidence  in  the  overall  product  suite  and  the 
adoption  rate  will  likely  be  low. 
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MAY  USER  TESTS 


OVERVIEW 

This  section  contains  an  overview  of  the  background  of  the  participants  for  the  portal  study.  This 
research  was  included  24-minute  participant  sessions.  Table  C-2  shows  a  breakdown  of  participants 
by  education  level  and  type. 

The  study  participants  were  characterized  according  to  the  following  specifications: 

•  Gender:  Both  male  and  female 

•  Experience:  Less  than  a  year  up  to  16+  years 

•  Number  of  Concurrent  Projects:  1-10 

•  Education:  At  least  bachelor’s  degree  ranging  to  post-doctorate  work 

•  Codes:  42,  52,  53,  56,  59,  71,  72,  and  81 


Table  C-2.  Participants  by  education  level  and  type. 


Highest  Degree  Earned 

Area  of  Study 

Bachelor’s  degree 

Computer  Science 

Bachelor’s  degree 

Electrical  Engineering 

Master’s  degree 

Business  Administration 

Bachelor’s  degree 

no  information  available 

Bachelor’s  degree 

Computer  Engineering 

Master’s  degree 

Systems  Engineering 

Master’s  degree 

Systems  Engineering 

Post-graduate  degree 

Organizational  Change  and  Development 

Bachelor’s  degree 

Information  Systems 

Post-doctoral  degree 

Neuroscience 

Master’s  degree 

Information  Systems 

Post-doctoral  degree 

Oceanography 
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METHODOLOGY 


OVERVIEW 

This  third  session  of  usability  testing  used  four  steps  to  evaluate  the  portal:  pretest  data  collection, 
protocol  script,  task  analysis,  and  a  wrap-up.  The  four  steps  are  detailed  in  this  section.  Pre-session 
data  collection  was  conducted  both  online  and  through  email.  Each  test  session  consisted  of  focused, 
task-based  analysis  following  a  think-aloud  protocol  designed  to  help  users  share  their  experiences 
verbally  during  the  experience.  Each  test  took  approximately  40  minutes.  These  methods  helped 
researchers  evaluate  how  well  users  could  complete  the  tasks,  and  provided  insight  as  to  common 
difficulties,  distractions,  or  miscommunications  encountered  by  the  participants. 

STEP  1 :  PRETEST  DATA  COLLECTION 

Prior  to  testing,  participants  answered  some  basic  questions  concerning  their  experience  as  a 
project  manager.  Participants  were  asked  the  number  of  projects  they  typically  work  on,  and  the 
number  of  projects  they  typically  manage  (see  Figure  C-l). 

STEP  2:  PROTOCOL  SCRIPT 

Each  test  session  began  with  one  researcher  reading  the  test  protocol  script  and  obtaining 
affirmative  consent  to  the  audio  and  video  recording.  As  a  part  of  this  protocol,  the  researcher 
discussed  the  goals  for  the  test  session,  expectations  for  the  participant,  and  encouragement  to  follow 
a  “talk-aloud”  protocol,  that  allowed  the  participant  to  talk  through  the  interface  so  researchers  could 
gain  insight  into  the  user  experience.  As  a  part  of  this  encouragement,  the  researcher  played  an 
example  video  of  other  users  employing  this  protocol.  The  entire  script  is  shown  in  Figure  C-2. 
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Figure  C-l  shows  the  survey  template  used  during  the  research  conducted  May- June  2015. 


_ Participant  Code  _ SSC-Pacific  Code 

Please  select  your  highest  degree  earned 

□  High  school 

□  Bachelors 

□  Post  graduate 

□  Masters 

□  Doctorate 

□  Post  doctorate 

Area  of  study: _ 

How  many  years  have  you  spent  in  government  service? 

_ years 

How  long  have  you  been  a  project  manager? 

□  Less  than  1  year 

□  1-5  years 

□  6-10  years 

□  1 1-15  years 

□  16  +  years 

On  average,  how  many  projects  do  you  typically  manage  at  the  same  time?  Please  include  any 
projects  where  you  are  listed  as  the  project  manager  or  IPT  Lead 

□  1  project 

□  2-3  projects 

□  4-5  projects 

□  6-10  projects 

□  11+  projects 

How  often  do  you  delegate  project  management  tasks  like  filling  out  the  WAT  or  PIT  to  another 
person? 

□  Always 

□  %  of  the  Time 

□  Half  of  the  Time 

□  lA  of  the  Time 

□  Never 

Have  you  ever  completed  a  PMP? 

□  Yes 

□  No 

Figure  C-1 .  Survey  template. 
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Figure  C-2  shows  the  protocol  template  used  during  the  research  conducted  May- June  2015. 


Thank  you  for  making  time  in  your  schedule  for  us! 


My  name  is _ .  I’m  a  Human  Factors  Engineer  in  the  UCD&E  group  and  we’re  helping  design  a 

suite  of  tools  to  help  support  our  project  managers. 

This  is _ s/he’ll  be  taking  notes  throughout  our  session. 


Today  we’d  like  to  show  you  a  series  of  digital  sketches  of  what  a  new  center  process  tool  might  look 
like.  These  are  early  designs  so  some  parts  of  them  may  be  incomplete.  There  will  be  elements  that 
have  fairly  good  detail,  and  there  will  also  be  elements  that  will  be  completely  missing. 

During  this  session,  we  will  be  taking  notes  (and  recording)  in  order  to  make  a  record  of  our  session. 
These  notes  and  recordings  are  only  for  our  reference.  They  are  simply  materials  we  will  use  as  we 
revise  our  design  and  update  User  Requirements.  None  of  the  information  you  share  with  us  will  be 
attributed  to  you  and  will  be  anonymized  in  any  supporting  documentation  or  presentations  we  develop 
as  a  result  of  this  session. 

While  we  look  at  these  pages  together,  I  want  to  remind  you  we're  always  testing  designs  and 
processes,  never  you.  You  are  giving  us  valuable  insights  that  will  allow  us  to  help  users  like  you.  So  be 
as  detailed  and  honest  as  you  can,  and  try  not  to  be  nervous.  Remember  this  is  not  a  test  of  your 
abilities,  but  a  test  of  the  page  design.  If  something  is  not  clear,  that’s  OUR  fault. 

What  we  are  going  to  do  today  is  called  a  “think-aloud  protocol”.  What  we  will  do  is  give  you  a  scenario 
similar  to  something  that  might  actually  happen  in  real  life,  with  a  list  of  activities  to  perform.  We  would 
like  you  to  complete  the  activities  as  best  as  you  can  with  the  interface  we  are  looking  at.  During  this 
time,  we  need  you  to  think  aloud  and  verbally  walk  us  through  your  reactions  and  decision-making.  I 
need  to  record  these  reactions  to  ensure  I  fully  understand  your  answers  and  to  help  me  faithfully 
represent  your  needs  when  we  present  our  findings.  Just  try  to  say  everything  that  comes  into  your 
head,  your  reactions,  your  insights,  what  you’re  looking  for...  anything  and  everything  helps. 

To  help  you  better  understand  what  we  need  from  you,  let’s  watch  a  brief  video  to  see  how  another  user 
did  this  successfully. 

**Show  video** 

The  most  important  part  about  your  participation  in  this  session  is  that  you  consider  each  task  carefully 
and  provide  me  with  as  much  detail  as  you  can. 

Finally,  it’s  important  for  you  to  know  that  you  may  stop  this  session  at  any  point.  Similarly,  if  I  ask  you  to 
do  something  you  cannot  do,  for  any  reason,  simply  tell  me  and  I’ll  move  on.  It’s  important  that  you  feel 
comfortable  throughout  this  process.  If  you  want  to  stop  at  any  time,  just  let  me  know. 

Before  we  begin,  do  you  have  any  questions  or  concerns  about  this  process? 

Are  you  ready  to  begin? 

Figure  C-2.  Protocol  template. 
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STEP  3:  TASK  ANALYSIS 


Before  the  test,  each  participant,  identified  only  by  a  participant  ID  number,  was  randomly 
assigned  an  order  in  which  to  complete  two  tasks. 

Two  scenario  cards  were  used  in  the  task  analysis  test: 

1 .  Task  CPX  “Stormcloud”  scenario  was  a  task  in  which  the  participant  was  asked  to  start  a 
project  and  add  personnel  to  the  project  while  assigning  roles.  The  CPX  Stormcloud 
scenario  card  used  for  this  test  is  shown  in  Figure  C-3.  On  the  CPX  Stormcloud  scenario 
card,  the  portal  test  participant  was  told  to  perform  as  a  project  manager  for  a  hypothetical 
project  called  “Stormcloud”.  The  participant  was  asked  to  perform  two  tasks:  (1)  create  an 
entry  for  a  new  management  project,  Stormcloud  in  the  portal  software,  and  (2)  enter  the 
name  of  one  person  to  work  with  them  on  the  Stormcloud  project.  At  the  beginning  of  the 
CPX  task,  the  first  scenario  card  CPX  Stormcloud  was  read  aloud. 


Scenario: 

You  work  at  SSC  Pacific  on  multiple  projects.  A  new  tool 
for  managing  center  processes,  called  the  Project 
Process  Portal  has  just  been  released.  In  this  scenario, 
you  are  being  asked  to  use  this  tool  to  complete  several 
important  activities. 


CPX  (Stormcloud) 

Your  branch  head  informs  you  that  vou  are  going  to  be  the  project  manager 
for  a  new  project  called  'Stormcloud'.  He  asks  you  to  establish  Stormcloud  in 
the  Project  Process  Portal  by  close  of  business  today.  Additionally,  you  have 
identified  one  person  who  will  be  working  for  you  on  this  project:  Sally 
Hansen.  You  need  Sally  to  help  you  manage  the  Stormcloud  team,  including 
making  resource  and  tasking  decisions. 


Activity  1:  Use  the  Project  Process  Portal  to  start  a  new  project  called 
"Stormcloud." 


Activity  2:  Assign  Sally  to  the  Stormcloud  project  as  someone  with  authority 
to  manage  tasks  and  resources  within  the  Project  Process  Portal. 


Figure  C-3.  Task  CPX  Stormcloud  Scenario  card. 


2.  The  other  task,  JPIY,  shown  in  Figure  C-4,  and  MJPY,  shown  in  Figure  C-5,  were  given 
the  project  name  “Viceroy”.  These  scenario  cards  asked  the  participant  to  join  a  project, 
either  through  an  invitation,  or  solicitation  of  an  invitation.  The  participant  was  tasked  to 
find  and  check  off  some  basic  information  about  the  project’s  status  and  download  a  roster 
of  the  personnel  associated  with  the  project  using  the  portal  software.  The  assignment  of 
JPIY  or  MJPY  was  random. 
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J PI Y  (Viceroy) 

Richard  Burl,  your  new  project  manager,  has  just  sent  you  an  email: 

Activity  1:  Using  the  link  in  the  email,  accept  the  invitation  to  become  a 
team  member  on  project  Viceroy. 

Activity  2:  Determine  whether  or  not  the  PM  has  finished  submitting  the 
work  acceptance  package  for  Viceroy. 

Activity  3:  Create  a  distribution  list  for  Viceroy  by  downloading  a  roster  of 
everyone  on  the  project. 


Figure  C-4.  Task  JPIY  Viceroy  Scenario  card. 


MJPY  (Viceroy) 

Your  branch  head  asks  you  to  provide  support  to  project  manager 
Richard  Burl  on  the  Viceroy  project.  She  asks  that  you  request  access 
to  the  project  information  using  the  the  new  Project  Process  Portal. 
She  gives  you  a  link  to  the  portal. 

Activity  1:  Using  the  Project  Process  Portal,  request  access  to  the 
Viceroy  Project. 

Activity  2:  Determine  whether  or  not  the  PM  has  finished  submitting 
the  work  acceptance  package  for  Viceroy. 

Activity  3:  Create  a  distribution  list  for  Viceroy  by  downloading  a  roster 
of  everyone  on  the  project. 


Figure  C-5.  Task  MJPY  Viceroy  Scenario  card. 

After  each  task  was  completed,  the  participant  relayed  his/her  completion  status  to  the  researcher 
and  the  participant  was  evaluated  on  their  success  in  completing  the  tasks. 

STEP  4:  WRAP-UP 

Once  the  tasks  were  completed,  users  were  permitted  to  explore  the  interface  at  their  own  pace,  if 
desired.  The  research  team  also  answered  questions  about  the  goals  of  the  research,  the  state  of  the 
tool,  and  various  other  curiosities  presented  by  users.  Researchers  then  stopped  the  recordings, 
dismissed  the  participant,  and  proceeded  to  use  individual  and  group  notes  to  capture  the  lessons 
learned  during  the  session  for  later  use  in  meetings  and  reports. 
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FINDINGS 


OVERVIEW 

This  section  contains  findings  found  during  testing  for  this  report.  The  section  describes  users’ 
overarching  expectations  for  a  future  tool  or  project  management  space,  and  disappointments  with 
current  tools  and  Center-level  project  management  practices.  During  this  study,  the  UCD&E  team 
identified  many  separate  but  related  needs  that  turned  up  repeatedly  in  the  participants’  user  stories, 
both  in  open  interviews  and  during  the  task-oriented  walkthroughs.  The  results  are  grouped  into 
categories  for  further  discussion. 

SUCCESSES 

A  major  positive  feature  of  the  portal  is  its  simplicity.  Numerous  participants  commented  about  the 
clean,  uncluttered  look  of  the  portal,  indicating  preference  for  this  style  of  interface  over  the  cluttered 
look  and  feel  of  many  other  work  tools.  In  particular,  the  portal  received  high  marks  for  its 
straightforward  navigation  and  learnability. 

Participants  said  the  following  about  the  portal  interface: 

“Nice,  simple,  easy  to  understand.” 

“It  feels  clean,  it  feels  pretty  straightforward  . . .  There’s  not  too  many  things  on  the  page, 
there’s  not  too  many  things  that  I  can  click  on  . . .  the  layout’s  pretty  good.” 

“I’m  just  hoping  that  by  helping  and  taking  a  little  bit  of  time  that  maybe  we  could  get  some 
really  good  software  so  we  can  all  do  our  jobs  a  lot  faster.” 

The  portal’s  current  front-page  design  leverages  a  clean-look  design  and  judicious  use  of 
whitespace  that  users  favored  in  previous  interviews.  Those  characteristics,  coupled  with  sensible 
navigation  and  good  communication  lead  to  a  highly  valuable,  friendly  user  experience.  One  user 
commented  that  the  navigation  seemed  “too  easy,”  unlike  his  previous  experiences  at  the  Center. 

Participants  said  the  following  about  other  experience  with  other  software  interfaces: 

“It’s  so  important,  ‘cause  there’s  so  many  hundreds  of  us  that  are  going  to  rely  on  this,  that 
we  want  it  to  be  user  friendly.  Currently  today,  the  software,  in  my  opinion  at  SPAWAR,  is 
not  very  user  friendly  ...  It’s  just  painful  ...  It’s  such  a  waste  of  time,  times  hundreds  or 
thousands  of  people  ...  I’m  just  hoping  we  can  save  a  little  time  here.” 

FAILURES 

The  biggest  failure  of  the  current  design  seems  to  be  the  confusing  way  visual  presentation  of  the 
PMBOK  activity  map  and  process  framework  data  to  users.  Even  without  asking  the  participants  to 
comment  on  them  directly,  feedback  from  users  is  uniformly  negative  towards  these  elements. 

Participants  said  the  following  about  the  presentation  in  the  portal  software  of  the  PMBOK  activity 
map  and  process  framework  data  to  users: 
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“This  is  much  more  complicated,  [audible  *oof*]  Initially  it’s  a  lot  of  information  up  front.” 

“I’m  not  sure  what  all  this  stuff  is  at  the  bottom,  all  these  bars  . . .  [laughs]  but  it  looks  like  a 
lot  of  information.” 

“They  give  me  this  process  flow  that  I  never — these  are  on  every  one  of  these  tools  and  I 
never  read  them  ‘cause  ...  I  don’t  know  what  they’re  trying  to  tell  me,  so  I  just  ignore  them.” 

Because  of  the  distracting  nature  of  the  process  framework  and  accompanying  PMBOK  activity 
map,  the  UCD&E  team  suggests  removing  them  from  the  portal.  If  the  product  owners  insist  on 
retaining  one  or  the  other,  we  suggest  retaining  the  process  framework,  while  de-emphasizing  the 
shapes  and  colors  to  show  users  that  it  is  a  reference  diagram,  and  not  an  active  page  navigation 
element. 

Alternatively,  placing  the  diagrams  in  a  more  inconspicuous  area  of  the  portal  is  acceptable.  In 
future  discussions  with  the  Process  Architecture  Improvement  Project  (PAIP)  team,  we  will  offer 
suggestions  that  touch  on  each  of  these  suggestions. 

One  participant  said  the  following  about  the  presentation  in  the  portal  software  of  the  PMBOK 
activity  map  and  process  framework  data. 

“I  know  you  guys  are  held  by  the  PM  guide,  or  whatever,  but  this  whole  [process  framework 
and  PMBOK  activity  map]  concept  is  just,  it’s  really  pathetic.  We  don’t  . . .  This  is  not  how  it 
works  [in  real  life].” 

THREE  USER  TYPES 

There  are  three  types  of  user  types  for  the  portal  software  that  are  discussed  in  this  section. 
Observing  the  task  paths  and  feedback  acquired,  it  seemed  as  though  each  of  the  users  fell  into  one  of 
three  loosely  defined  categories. 

1 .  Type  One  users:  The  first  category  is  that  of  older  software  users.  These  users  learned 
computer  skills  at  a  later  age.  They  do  not  scroll  through  interfaces,  they  expect  things  to 
be  immediately  apparent.  There  is  additionally  an  expectation  for  information  to  be 
presented  in  links  or  documents  with  clear  hyperlinks,  well  placed  throughout  the  interface. 
Users  doesn’t  expect  a  rich  user  experience.  They  want  very  little  from  the  portal  website. 
They  want  to  get  in,  find  what  is  needed,  and  to  get  out  of  the  portal.  Because  the  user 
doesn’t  expect  much  from  the  experience,  working  hard  to  get  what  needs  to  be  done  in  the 
software  is  not  an  option.  “Give  me  the  information,  and  let  me  get  on  my  way”  is  the 
mantra  of  these  users.  This  approach  tends  to  be  the  classical  way  senior  personnel 
generally  interact  with  the  portal  and  computers. 
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One  experienced  said  the  following  about  the  ease  of  use  of  the  portal  software: 

“It  would  be  nice  if  I  had  a  one-stop-shop,  whereby  I  put  the  information  in  once,  and  it  fills 
out  all  the  tools  that  the  Center  requires  instead  of  me  having  to  go  to  each  one,  so  hopefully 
something  like  this  would  be  able  to  enter  it  once  and  whatever  the  upper  management  wants 
to  slice  and  dice,  it’s  a  one-time-shop — I  don’t  want  to  have  to  do  multiple  things  as  I’m 
doing  right  now.” 

2.  Type  two  users:  These  users  are  mid-late-career  professionals.  They  are  an  experienced 
user  type  and  skew  on  the  older  side  of  test  participants.  They  do  not  have  as  much 
computer  experience  as  the  more  experienced  Type  One  users.  They  have,  however,  seen  a 
lot  of  different  programs,  and  a  lot  of  turmoil  throughout  their  career  working  with 
different  software  solutions.  These  users  inherently  don’t  trust  the  portal  website.  Because 
of  this  distrust,  the  users  constantly  need  reassurance  and  proactive  feedback  to  ensure  that 
what  they  are  expecting  the  software  portal  to  do  is  exactly  what  they  are  experiencing,  and 
vice  versa.  Because  these  type  of  users  have  experienced  a  lot  of  software  issues  in  older 
systems,  they  expect  problems  to  occur  with  any  new  software  they  use.  They  are  very 
patient  with  the  software  when  issues  do  occur.  They  wait  because  of  the  (expected) 
difficulties.  If  the  software  doesn’t  immediately  do  what  the  users  expect  it  to  do,  they  will 
wait  (often  unnecessarily)  before  trying  something  else.  These  users  also  prefer  as  little 
interaction  with  the  interface  as  possible. 

An  experienced  user  type  two  had  this  to  say  the  portal  software  and  having  too  many  options  in  one 
place. 

“That’s  the  biggest  frustration.  If  we  want  to  go  to  another  tool,  then  let’s  go  to  another  tool, 
let’s  not  add  another  tool  and  have  us  do  them  all!” 

3.  Younger  computer  users:  These  users  have  typically  “grown  up”  using  computers.  It  is 
second  nature  for  these  users  to  scroll  a  lot,  searching  and  scanning  for  cues  contained  in 
the  interface.  For  these  users,  immediate  feedback  is  paramount.  These  users  have  little 
patience  for  slow  or  unreliable  interfaces.  They  are  confident  in  their  ability  to  navigate 
and  intuitively  pick  up  on  navigation  cues.  They  expect  high  performance  and  advanced 
capabilities  from  the  portal.  These  users  will  test  the  limits  of  the  portal  navigation 
capabilities.  As  long  as  these  conventions  are  sound  and  properly  implemented,  these  users 
will  take  advantage  of  them  better  than  most.  These  users  expect  to  accomplish  a  lot  within 
the  portal,  but  are  against  spending  too  much  time  to  get  where  they  want  to  be  in  the 
portal  to  get  the  portal  to  do  what  they  need  it  to  do. 

Younger  users  had  this  to  say: 

“Make  the  site  powerful  and  quick  to  respond  to  commands.”  If  this  is  done,  we  feel  the  user 
will  appreciate  the  experience. 
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One  younger  user  had  this  to  say  regarding  having  more  capabilities  in  the  portal  software: 

“Having  more  capabilities  would  make  me  like  the  tool  more  because  it  gives  me  something  I 
could  maybe  use  on  a  day-to-day  basis,  rather  than  a  tool  that  I  just  go  into  every  three 
months  to  just  click  “go”  so  everyone  up  the  chain  is  happy.” 

“I  would  want  to  be  able  to  go  into  there  and  actually  assign  [supervisors,  branch  heads,  etc.  ) 
the  privilege  of  viewing  the  details  . . .  automatically  generate  a  report  . . .  that’s  such  a  great 
time  saver.” 

Normally,  it  can  be  a  bit  of  a  challenge  to  design  a  site  that  caters  equally  well  to  all  of  these  three 
types  of  users,  as  well  as  others  who  may  not  neatly  fit  into  these  categories.  The  common  threads  of 
simplicity,  straightforward  navigation,  and  increased  real-time  feedback,  however  will  go  a  long 
ways  towards  accomplishing  this  goal. 

OTHER  NOTES 

Users  had  some  general  confusion  with  the  top  navigation  links  and  the  left-hand  navigation  links 
in  the  project  dashboard  page.  Some  of  this  confusion  was  due  to  some  inactive  links;  these  kinds  of 
issues  will  naturally  lessen  as  the  portal  is  developed  further.  It  is  important  to  maintain  a  separation 
in  functionality,  except  as  specifically  called  out  in  the  design  specs  (“Create”  button,  etc.).  We  will 
work  closely  with  the  development  team  to  iron  out  these  issues.  When  successfully  implemented, 
the  portal  software  will  be  refreshing  and  delightful  for  users. 

One  portal  software  participant  said  the  following  when  trying  to  locate  the  project  management 
tracking  (PMT)#  function: 

“I  think  this  is  awesome  how  it  has  the  project  name  and  PMT#  in  the  top  left  [of  the 
dashboard]  so  you  always  know  [where  you  are  in  the  portal],  especially  when  you’re  in 
multiple  projects,  so  you  always  know  you’re  in  the  right  one.  I  think  that’s  cool.” 

Within  the  project  portal,  the  word  “administrator”  has  different  meanings  in  different  contexts.  In 
future  designs,  it  will  be  important  to  differentiate  between  system  administrator,  site  administrator, 
project  administrator,  and/or  team  administrator,  as  needed. 

An  additional  item  discovered  as  a  part  of  these  user  tests  concerns  the  process  of  navigating  to, 
and  using  the  project  team  roster  of  personnel.  There  is  a  strong  preference  for  users  to  navigate  to  a 
project-specific  roster  only  through  the  project  page  itself.  By  requiring  users  to  first  access  a  multi¬ 
project  personnel  roster  then  filter  that  to  obtain  a  project-specific  roster,  the  users’  mental  models  of 
how  the  interface  should  handle  this  situation  was  violated,  leading  to  confusion  and  distrust  in  the 
interface  to  give  the  correct  output.  As  originally  designed,  there  should  be  a  project-specific  roster 
link  from  within  the  project  dashboard  page;  this  button  is  currently  present,  but  not  yet  active.  Once 
this  is  fixed,  the  UCD&E  team  suggests  re-testing  this  item.  More  than  half  of  the  participants  tested 
had  significant  issues  with  its  functionality. 

Across  the  board,  virtually  every  participant  expressed  or  otherwise  indicated  the  desire  for  more 
comprehensive,  timely  feedback.  Equally  important  are  sufficient  feed-forward  mechanisms,  put  in 
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place  to  help  manage  users’  expectations  and  improve  navigability.  Hesitation,  insecurity,  and 
excessive  scanning  for  cues  are  all  symptoms  of  this  condition.  More  feedback  is  needed,  especially 
upon  completion  of  tasks  such  as  project  creation  or  responding  to  join  requests. 

One  portal  software  participant  said  the  following: 

“The  feedback’s  good  ...  I  know  who  [the  request  is]  sent  to.  I  like  that  the  feedback  is  there, 
in  case  you  click  on  the  wrong  one.” 
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A/B  STUDY 


OVERVIEW 

This  section  contains  findings  found  during  A/B  study  comparisons  done  for  this  report.  For  this 
study,  the  UCD&E  team  used  an  eight-question  survey  to  compare  four  competing  design  options, 
distributed  to  study  participants  by  the  online  polling  tool,  SurveyMonkey®.  The  purpose  of  the 
survey  was  to  generate  feedback  about  which  of  the  four  approaches  to  the  secondary  navigation 
users  deemed  “easier  to  use.”  The  UCD&E  team  identified  four  potential  design  configurations, 
mocked  them  up  and  paired  them  for  experiments.  The  orange  circled  functions  shown  in  Figure  C-6 
portray  the  same  function  displayed  in  different  configurations. 
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Figure  C-6.  Four  potential  design  configurations,  (mocked  up  for  comparison). 

Six  pairs  of  screenshots  were  randomly  displayed  to  each  participant  (AB,  AC,  AD,  BC,  BD,  and 
CD).  Users  were  asked  to  choose  their  favorite  from  two  displayed  options.  Users  selected  their 
favorites  among  all  six  pairs  presented. 


C-18 


Paired  screenshots  are  shown  in  Figure  C-7,  where  the  user  was  given  a  choice  of  two  screen 
configurations  and  asked  to  choose  which  was  easier  to  use. 


Project  Management  Tool  Suite-  Display  Options 


*  i.  Is  X  or  Y  easier  to  use? 

Q  Image  X  Q  Image  Y 


Prev 


Next 


Figure  C-7.  Paired  screenshots. 
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The  questionnaire  continued  by  asking  basic  demographic  questions  for  classification  purposes. 
The  questionnaire  is  shown  in  Figure  C-8. 


Figure  C-8.  Demographics  and  form  submission. 

The  polling  tool  required  users  to  make  a  selection  between  the  pairs  shown,  but  did  not  require 
answers  to  the  demographic  questions  so  their  answers  could  be  counted  in  the  overall  totals. 

The  UCD&E  team  advertised  this  survey  to  participants  through  the  Fusion®  website,  by  members 
of  the  Project  Manager  Council,  and  by  word-of-mouth  in  the  various  branches  to  reach  the  widest 
audience  possible  with  the  least  overhead. 

The  UCD&E  team  wanted  to  keep  the  cost  of  this  study  low  for  two  reasons: 

1.  The  team  opted  to  add  this  study  to  the  scope  of  work  agreed  to  help  address  a  complex 
issue  that  needed  solving;  the  cost  of  recruiting,  vetting,  and  assigning  participants  to  this 
study  was  not  included  in  the  original  budget. 

2.  The  team  wanted  to  provide  a  feedback  mechanism  to  the  broadest  population  possible 
without  incurring  excessive  costs,  yet  ensuring  open  opportunity  for  more  members  of  the 
Center  at  large  to  contribute  feedback  to  the  overall  PAIP  project. 

RESPONSE  AND  DEMOGRAPHICS 

Response  Rate:  Seventy-one  people  completed  the  entire  survey;  sixty-nine  answered  the 
demographics  portion,  volunteering  their  codes  participation  to  count  distribution.  Of  those 
responses,  Departments  5.0,  6.0,  7.0,  8.0  and  Code  H  were  all  represented,  with  Department  5.0 
comprising  the  majority  (76%)  of  participants.  (See  the  chart  provided  in  Figure  C-9). 
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Although  the  survey  asked  participants  to  identify  their  department,  the  mix  of  responses  ranged 
from  department  codes,  to  competency  codes,  to  5-digit  branch  codes.  For  consistency,  we  refer  to 
the  participants’  reports  of  the  work  unit  they  are  associated  with  as  “competency  codes.” 


Figure  C-9.  Participant  reports  by  competency  code  (percentage). 

Sixty-three  of  the  7 1  participants  responded  to  this  survey  with  demographic  information  about 
their  project  management  experience.  Of  those,  70%  had  some  degree  of  project  management 
experience,  with  the  greatest  number  of  participants  without  experience  coming  from  Department 
5.0. 

Figures  C-10  and  C-l  1  detail  project  management  experience  of  portal  users  for  participants  in  two 
ways.  Figure  12  provides  an  comparison  of  participants’  project  management  experience  that 
participated  in  the  portal  study  shown  in  percentages. 
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Participant  Experience  in  Project  Management 

■  Less  than  a  year  *1-5  years 

■  6-10  years  ■  11-15  years 

■  1 6+  years  ■  I'm  not  a  Project  Manager  or  IPT  Lead 

Project  Management 
Experience  (n = 63) 

Figure  C-10.  Participant  project  management  experience  by  percentage. 
Figure  C-ll  details  participants’  project  management  experience  by  competency  code. 


3%  19%  11%  13%  24%  30% 


Figure  C-1 1 .  Project  management  experience  by  competency  code. 
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COMPARISONS 


The  UCD&E  team  performed  a  trend  analysis  of  portal  test  participants  looking  for  patterns  in  the 
screen  configuration  designs.  Figure  C-12  shows  portal  users’  design  selections  preference  ratings  for 
A,  B,  C,  and  D  alternative  screen  designs.  Figure  C-12  shows  the  two  most  preferred  designs  were  A 
and  D,  consistently  over  B  and  C.  Each  design  configuration  presented  a  choice  of  two  portal  screen 
design  configurations.  Four  design  configurations  were  presented.  Figure  C-12  shows  the  popularity 
of  design  configurations  by  percent  age  based  on  portal  users’  choices. 

N  =  71  PER  "A  BB  BC  BD 


PAIR  1  (AB)  -A  PAIR  2  (CA)  -A  PAIR  3  (AD)  -D  PAIR  4  (BC)  -C  PAIR  5  (DB)  -D  PAIR  6  (CD)  -D 


Figure  C-12.  Software  configuration  reference  data  individual  comparisons. 

Design  configuration  A  =  blue 
Design  configuration  B  =  red 
Design  configuration  Z  =  green 
Design  configuration  D  =  purple 
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Figure  C-13,  highlights  PAIR  3  (A/D)  -D  which  was  favored  at  59%  over  PAIR  1  (AB)  -A  at  41% 
as  the  preferred  configuration. 


N  =  71  PER  ■  A  B  C  ■  D 


59% 


41% 


PAIR  1  (AB)  -A  PAIR  2  (CA)  -A  PAIR  3  (AD)  -D  PAIR  4  (BC)  -C  PAIR  5  (DB)  -D  PAIR  6  (CD)  -D  D 

Figure  C-13.  Preference  data  individual  comparisons  focus  on  PAIR  3  (AD)  -  D. 

Design  configuration  A  =  blue 

Design  configuration  D  =  purple 

Least  preferred  design  configurations  =  muted  gray 

The  UCD&E  team  performed  a  heuristic  evaluation  of  the  designs  and  concluded  that  design  A 
was  likely  insufficient  to  support  the  textual  density  of  the  content.  The  results  of  the  survey 
supported  this  conclusion,  coinciding  with  the  executed  design,  allowing  for  unfettered  presentation 
of  the  textual  content.  As  a  result  of  the  participant  input,  the  project  activity  navigation  scheme  was 
redesigned.  The  redesign  of  the  navigation  menu  improved  access  and  ease  of  use  of  the  navigation 
menu.  The  navigation  menu  that  was  formerly  on  the  upper  left  of  the  portal  screen  is  now  displayed 
in  the  center  workspace  as  shown  in  Figure  C-14. 


Navigation  menu  was  moved  to 
the  center  workspace 


JACOB  VIRALDO 

ra  SSC  PAC  BUSINESS  APPLICATIONS 


VWCOmt  JACOeWM root 


Project  Team 

I  ooBOt 


Figure  C-14.  Project  activity  navigation  scheme  (latest  design). 
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CONCLUSION 


Usability  tests  of  early  portal  iterations  conducted  in  May  2015  continued  to  show  successes, 
especially  in  navigation,  interface  design,  and  user  task  performance.  Users  expressed  delight  with 
the  look  and  feel  of  the  portal  and  demonstrated  that  they  could  perform  basic  initial  project 
management  tasks  through  the  interface  with  few  issues.  Going  forward,  the  UCD&E  Branch  will 
continue  to  refine  these  designs,  leveraging  the  lessons  learned  from  this  round  of  testing.  The 
ultimate  goal  is  to  improve  an  interface  design  that  co-locates  tools  and  resources  required  for  project 
management  in  an  easily  navigable  format. 

Participants  said  the  following  about  the  downloading  the  roster: 

“This  is  fun!” 

Future  designs,  recommendations,  and  briefings  by  the  User-Centered  Design  &  Engineering  Team 
will  address  the  following: 

•  Removing,  de-emphasizing,  or  reducing  distractions  caused  by  PMBOK  Activities  and  Process 
Framework  Overview 

•  Improving  placement  and  timeliness  of  feedback  and  feed-forward  mechanisms 

•  Using  clear  and  consistent  terminology  to  increase  recognition,  learnability,  and  user  trust 

The  User-Center  Design  &  Engineering  Branch’s  recommendations  are  based  on  carefully 
designed  usability  tests  and  user  research,  as  well  as  ongoing  heuristic  evaluations.  As  user  research 
and  usability  testing  progress,  recommendations  will  be  shared  with  the  development  team  and  others 
across  the  project’s  organization. 

Participants  said  the  following  about  using  the  portal  to  create  a  project: 

“Pretty  easy!” 

The  UCD&E  team  is  confident  that  its  iterative  design-test-revise-develop  model  is  continuing  to 
produce  many  benefits  as  it  helps  reveal  user  expectations  and  flaws  in  the  product’s  design. 
Additionally,  with  attention  to  UCD&E’s  recommendations,  the  Center’s  Project  Portal  User 
Community  will  ultimately  be  well  served  with  a  consistent  and  professional  user  experience. 
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APPENDIX  D 


EXECUTIVE  SUMMARY 

As  part  of  the  effort  in  supporting  the  Space  and  Naval  Warfare  Systems  Center  Pacific  project 
portal,  the  User-Centered  Design  and  Engineering  (UCD&E)  team  conducted  three  rounds  of  rapid 
user  testing  to  quickly  and  efficiently  identify  and  correct  the  major  interface  impediments  and 
system  bugs  in  the  existing  design.  As  part  of  this  action,  several  key  interface  elements  and  system 
bugs  were  identified,  documented,  and  corrected.  This  report  discusses  those  bugs. 

The  most  significant  issues  to  address  and  correct  include: 

1 .  Shortening  computer  wait  times  between  tasks 

2.  Correcting  issues  with  the  personnel  lookup  tool 

3.  Debugging  and  implementing  fixes  for  several  bugs  discovered  during  testing 

Most  of  elements  the  UCD&E  team  tested,  beyond  issues  found,  are  documented  in  this  test  report. 
Tested  issues  tested  extremely  well.  Portal  users  successfully  completed  assigned  tasks.  Any  issues 
found  during  testing  are  in  this  report.  Three  rounds  of  testing  proved  sufficient  to  capture  most  of 
users’  issues.  The  test  pace  was  fast  enough  to  prevent  unnecessary  work  for  the  development  team. 
The  pace  was  moderate  enough  to  allow  updates  to  be  added  for  smaller  groups. 
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INTRODUCTION 


PREVIOUS  PORTAL  TEST  LESSONS  LEARNED 

The  project  portal  user  design  is  supported  by  the  user  centered  design  and  engineering 
User-Centered  Design  and  Engineering  (UCD&E)  team  at  Space  and  Naval  Warfare  Systems  Center 
(SSC  Pacific).  The  team  previously  supported  the  software  development  team  by  conducting 
extensive  user  testing  sessions.  These  sessions  were  conducted  April-May,  2015.  The  sessions 
provided  a  wealth  of  information  to  researchers,  offering  insight  on  users’  needs,  expectations, 
problems,  tendencies,  mental  models,  and  desires  for  future  tool  capabilities. 

RAPID  USER  TESTING  CYCLE 

As  the  project  neared  completion,  it  was  necessary  to  test  users  in  smaller  batches  and  in  shorter 
intervals.  This  technique  is  referred  to  as  rapid  iterative  testing  and  evaluation  or  rapid  user  testing. 

Rapid  user  testing  is  a  tool  used  often  by  UCD&E  in  situations  where  a  research  team  wants  to 
speed  up  the  iterative  process  of  “test-design-code-repeat”  shown  in  Figure  D- 1 . 
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Figure  D-1.  Rapid  user  testing  iterative  process. 

In  rapid  user  testing,  changes  to  the  user  interface  are  made  as  soon  as  a  problem  is  identified  and  a 
solution  is  clear.  These  changes  can  usually  be  done  in  a  single  day  with  minimal  investment  in  time, 
equipment,  or  user  participation.  During  portal  testing,  the  UCD&E  team  supported  the  rapid 
development  schedule  assigned  by  the  development  team  by  using  rapid  user  testing.  Rapid  user 
testing  enabled  the  UCD&E  team  to  meet  tight  schedule  demands. 

WHY  RAPID  USER  TESTING? 

Key  elements  of  rapid  user  testing  include: 
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•  Uses  fewer  participants  than  traditional  user  testing,  generally  three  to  six  participants  per 
test.  Despite  a  smaller  sample  size,  this  feedback  identifies  most  major  issues  in  user- 
centered  design  tests. 

•  Can  be  completed  in  one  day.  With  fewer  participants,  the  time  investment  is  minimal, 
and  scheduling  issues  are  minimized. 

•  Findings  are  reported  quickly.  Emailed  summaries  of  reported  findings  are  timely  and 
more  useful  than  formalized  reports,  especially  near  the  end  of  the  development  process. 

•  Due  to  the  smaller  sample  size  and  quick  turnaround,  costs  are  significantly  lower  than 
those  associated  with  traditional  user  testing.  When  a  project  is  on  a  tight  deadline,  with 
limited  funds,  rapid  user  testing  is  a  very  effective  means  of  user  testing. 

•  Overall  interval  time  (from  one  test  to  another)  is  shorter.  Work  needed  between  tests  is 
minimized,  allowing  the  whole  team  (including  development  and  management)  to  focus 
efforts  on  implementing  the  recommendations  and  improving  the  product. 

In  the  UCD&E  Branch,  rapid  user  tests  found  particularly  useful  for: 

•  Testing  ideas  quickly 

•  Validating  concepts  and  initial  designs 

•  Providing  new  insights 

•  Highlighting  areas  that  may  require  further  testing 

•  Settling  internal  disputes  quickly 

•  Preventing  project  teams  from  heading  down  the  wrong  path 

•  Informing  the  next  steps  of  the  design  and  direction  of  the  project 

To  keep  up  with  the  fast  pace  of  technological  development,  the  design  process  iterations  need  to 
happen  faster,  while  still  producing  high-quality  user  feedback,  which  is  the  role  of  rapid  user  testing. 
Despite  a  small  participant  sample  size,  rapid  user  testing  can  give  project  teams  the  direction  they 
need  to  pursue  or  discard  a  design,  and  improve  tool  functionality  and  user  experience. 
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METHODOLOGY 


OVERVIEW 

This  section  contains  methods  used  for  the  portal  study  rapid  user  testing.  Over  three  rounds  of 
rapid  user  testing  participants  were  asked  to  perform  a  series  of  tasks  within  a  limited  project  portal 
website  with  partial  functionality. 

The  tasks  chosen  for  each  round  were  based  on  several  factors: 

•  Desire  to  test  recently  enabled  functions 

•  Need  for  exploring  and/or  isolating  issues  found  during  previous  experiments 

•  Help  to  make  design  decisions  between  unsettled  options 

As  such,  at  various  points,  users  were  asked  to  perform  a  series  of  tasks  given  on  a  series  of  the 
Rapid  User  Testing  Scenario  task  cards  found  in  Figures  D-2  through  D-8,  including: 

•  Logging  in  with  CAC 

•  Creating  a  project 

•  Changing  a  project  logo 

•  Searching  for  personnel 

•  Adding  project  team  members 

•  Assigning  roles 

•  Editing  roles 

•  Downloading  project  rosters 

•  Opening  csv  roster  in  Excel® 

•  Marking  completion  of  tasks  in  project  dashboard 

•  Navigating  within  site 

•  Navigating  to  Project  Information  Tracking  tool  (PITT) 

Figure  D-2  shows  the  Scenario  task  cards  used  during  rapid  user  testing  scenario  testing  conducted 
October  1,  2015. 


Scenario: 

You  work  at  SSC  Pacific  on  multiple  projects.  A  new  tool 
for  managing  center  processes  called  the  Project  Portal 
has  just  been  released.  In  this  scenario,  you  are  being 
asked  to  use  this  tool  to  complete  a  few  important 
tasks. 

Figure  D-2.  Scenario  task  cards. 

Figure  D-3  shows  New  Project  cards  used  during  rapid  user  testing  scenario  testing  conducted 
October  1,  2015. 
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New  Project 

Your  branch  head  informs  you  that  you  are  going  to  be  the  project  manager 
for  a  new  engineering  project.  He  asks  you  to  establish  the  project  in  the 
Project  Portal  by  close  of  business  today. 

1:  Open  the  Project  Portal  and  login  with  your  CAC  by  navigating  to  this  link: 
https://wls-qa.sscpac.nmci.navy.miI/projectportal/#/ 

2:  Begin  initiation  on  a  new  project  in  any  manner  you  choose,  including 
choosing  a  name  and  nickname  for  the  project. 

3:  Navigate  back  to  the  main  Project  Portal  landing  page. 


Figure  D-3.  New  Project  cards. 

Figure  D-4  shows  update  Project  Roster  cards  used  during  rapid  user  testing  scenario  testing 
conducted  October  1,  2015. 


Update  Project  Roster 

Following  any  method  you  wish,  return  to  the  project  you  just  created. 

1:  Edit  your  roles  in  the  project  you  just  created,  making  sure  you  have  the  dual 
roles  of  Project  Manager  and  Lead  Systems  Engineer. 

2:  Add  SSC-PAC  employee  Sarah  Hunt  as  System  Administrator. 

3:  Add  another  SSC-PAC  employee  Alan  Lemon  to  your  project,  with  any  roles  you 
choose. 

4:  Download  your  current  project  roster,  and  open  it  in  Excel  to  verify  its  contents 
Figure  D-4.  Update  Project  Roster  cards. 

Figure  D-5  shows  Complete  Project  Initiation  cards  used  during  rapid  user  testing  scenario  testing 
conducted  October  1,  2015. 


Complete  Project  Initiation 

As  new  Project  Manager  of  the  project  you  just  created,  you  need  to 
now  take  your  project  through  the  Project  Initiation  process. 

1:  Enter  the  Project  Dashboard. 

2:  Mark  each  of  the  steps  in  Project  I  nitiation  as  though  you  have 
already  finished  them. 

3:  Complete  the  activity  by  checking  your  progress  in  the  PITT. 

Figure  D-5.  Complete  Project  Initiation  cards. 

Figure  D-6  shows  the  Stormcloud  card  used  during  rapid  user  testing  scenario  testing  conducted 
October  1,  2015. 
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Stormcloud 

Your  branch  head  informs  you  that  you  are  going  to  be  the  project 
manager  for  a  new  project  called  "Stormcloud".  He  asks  you  to 
establish  Stormcloud  in  the  Project  Portal  by  close  of  business  today. 

Activity  1:  Use  the  Project  Process  Portal  to  start  a  new  project  called 
"Stormcloud." 

Activity  2:  Go  into  the  Stormcloud  project,  and  update  the  project  logo 
to  the  Stormcloud  icon  in  the  Project  Icons  folder  on  your  desktop. 


Figure  D-6.  Stormcloud  cards. 

Figure  D-7  shows  Viceroy  cards  used  during  rapid  user  testing  scenario  testing  conducted  October 
1,  2015. 


Viceroy 

Following  any  method  you  wish,  establish  another  project  in  the  Project 
Portal 

Step  1:  Name  this  project  Viceroy. 

Step  2:  Return  to  the  Project  Portal  landing  page. 

Step  3:  Change  the  project  icon  to  the  Viceroy  icon,  as  before. 

Figure  D-7.  Viceroy  cards. 

Figure  D-8  shows  Project  3  cards  used  during  rapid  user  testing  scenario  testing  conducted 
October  1,  2015. 

Project  3 

Following  the  instructions  given  orally,  establish  another  projectin  the 
Project  Portal. 

Step  1:  Name  this  project. 

Step  2:  Return  to  the  Project  Portal  landing  page. 

Step  3:  Change  the  project  icon  to  one  of  your  choice. 

Figure  D-8.  Project  3  cards. 


D-9 


As  users  stepped  through  the  cards,  the  UCD&E  team  recorded  the  sessions  for  easier  data 
collection  in  post-session  analysis.  All  issues  found  were  immediately  analyzed,  documented,  and 
sent  out  through  email  to  the  development  team  with  recommended  changes  to  implement.  Two 
informal  wrap-up  test  result  reports  were  provided  through  email  to  the  development  team  and 
Process  Architecture  Improvement  Project  (PAIP)  team  in  August. 

TEST  ENVIRONMENT 

The  test  configuration  used  for  the  three  rounds  of  rapid  user  testing  consisted  of: 

•  A  laptop  and  mouse 

•  Internet  Explorer  10  and  Camtasia  Video  Capture®  Software  for  post-test  analysis. 

•  An  additional  monitor  to  mirror  (clone)  the  screen  used  by  the  participant,  which  allowed 
researchers  to  view  tasks  as  they  were  performed. 

All  testing  took  place  in  the  UCD&E  facilities  at  SSC  Pacific.  The  protocol  for  all  rounds  of  rapid 
user  testing  is  included  in  Appendix  B . 
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USERS 


OVERVIEW 

This  section  contains  an  overview  of  the  background  of  the  participants  for  the  portal  study. 

Participants  were  chosen  for  the  three  rounds  of  rapid  user  testing  through  three  methods: 

1 .  An  open  call  to  personnel  who  wanted  to  participate 

2.  Users  who  had  previously  wanted  to  participate  but  had  been  unable  to  fit  into  our  testing 
schedule 

3.  A  convenience  sample  of  nearby  interested  SSC  Pacific  employees. 

DEMOGRAPHICS 

The  chart  in  Figure  D-9  shows  the  percentage  of  employee’s  participations  in  the  rapid  user  tests 
by  code.  The  distribution  is  based  on  three  rounds  of  rapid  user  testing. 
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FINDINGS 


OVERVIEW 

This  section  contains  findings  from  the  rapid  user  testing  conducted  October  1,  2015. 

RAPID  USER  TESTING  FINDINGS 
Create  a  Project  Functionality 

Users  who  attempted  to  create  a  project  using  methods  provided  in  the  project  portal  interface  were 
successful.  Every  user  completed  the  task,  with  minimal  delays  or  confusion,  and  expressed 
confidence  in  repeating  the  task  in  the  future.  There  were  no  user-experience-related  mistakes.  Clear 
portal  definitions  and  instructions  led  to  successful  user  performance.  The  Input/Output  screen 
transitions  and  graphic  user  interface  wizards  performed  according  to  users’  expectations.  Users 
could  navigate  successfully  through  portal  software  solution  paths  and  could  complete  tasks  in  every 
instance.  Users  appreciated  the  ease  and  forethought  designed  into  the  portal  as  a  whole,  and  the 
straightforward  nature  of  the  graphic  user  interfaces. 

Uploading  and  Downloading 

Tasks  such  as  uploading  an  image  (for  use  as  the  project  logo)  and  downloading  data  to  Excel® 

(for  example,  a  project  roster)  were  successful.  Because  the  portal  follows  basic  conventions,  the 
resulting  interactions  met  users’  expectations  and  were  completed  without  errors.  The  only 
difficulties  that  occurred  were  related  to  minor  interface  issues  related  to  screen  refresh  rates  in  the 
portal.  In  each  instance,  a  project  logo  didn’t  appear  in  the  project  dashboard  after  upload;  the  user 
had  to  refresh  the  page  to  see  the  change.  We  suggested  to  the  development  team  that  that  refresh  is 
done  automatically  to  allow  the  user  to  see  the  result  of  their  actions  instantaneously. 

Intermittently  Long  Lag  Times 

Most  of  the  user  tests  had  few  technical  issues,  but  one  in  particular,  took  much  longer  than  the 
others.  One  test  participant  experienced  an  extraordinarily  long  lag  in  updating  the  project  roster  that 
resulted  in  the  user  downloading  an  incomplete  roster  (the  roster  was  missing  one  name). 

The  interactions  and  loading  times  of  that  one  test  are  listed  as  follows: 

•  Logging  in  with  CAC,  (from  browser  entry  to  portal  appearing):  60  seconds 

•  Clicking  on  “My  Teams”  until  roster  appeared  on  page:  34  seconds 

•  Searching  for  a  name  in  the  personnel  search  (from  click  to  result  appearing):  13  seconds 

•  Clicking  “Assign”  for  roles  until  the  name  appeared  in  the  roster  list:  22  seconds 

•  Searching  for  another  name:  10  seconds 

•  Assigning  roles,  and  appearing  in  roster:  22  seconds 

•  Clicking  on  “My  Teams”  again:  37  seconds 

•  Searching  for  another  person:  10  seconds 

•  Searching  for  another  person:  10  seconds 
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One  of  these  lags  caused  the  incorrect  download  shown  in  the  interactions  and  loading  times  list 
above.  For  unknown  reasons,  this  particular  user  had  longer  lag  times  than  any  of  the  other  users  tests 
conducted  during  these  sessions.  Some  of  the  other  tests  had  varying  lag  times,  though  they  were 
shorter  than  those  times  documented  above.  The  lag  times  were  not  always  present.  One  participant 
experienced  the  opposite,  nearly  every  interaction  with  the  portal  was  instantaneous.  The  UCD&E 
team  recommended  to  the  development  team  that  it  look  into  this  issue  and  take  actions  to  keep  wait 
times  as  short  as  possible. 

Personnel  Search  issues 

When  searching  for  a  name  in  personnel  search,  the  field  called  out  to  do  this  is  “Last  Name”  then 
“First  Name”.  The  labeling  of  these  boxes  lies  within  the  boxes  themselves  and  appears  before  any 
text  is  entered  in  the  field.  It  also  disappears  after  the  user  clicks  on  them.  One  user  had  difficulty 
noticing  these  labels,  and  kept  entering  the  desired  first  name  in  the  “Last  Name”  field,  and  the 
desired  last  name  in  the  “First  Name”  field.  When  the  search  returned  no  results,  the  user  was 
frustrated.  Additional  user  testing  is  recommended  to  determine  if  this  issue  is  widespread  among  the 
user  population,  or  an  isolated  occurrence. 

The  search  directory  results  appearing  in  the  portal  currently  have  no  order  associated  with  them. 
Users  expect  a  clear  order  in  their  search  results,  generally  alphabetical.  The  UCD&E  team 
recommended  to  the  development  team  to  adopt  a  default  configuration  of  “alphabetical  order  by  last 
name”  to  display  found  names. 

The  personnel  database  used  to  look  up  people  by  name  is  incomplete.  It  contains  many  names 
without  emails  or  phone  numbers.  Whenever  one  of  those  people  creates  a  project,  the  project 
encounters  numerous  errors.  Users  with  incomplete  records  are  prevented  from  being  added  to  a 
project  within  the  portal.  The  reasons  for  this  are  unclear  at  this  point. 

The  UCD&E  team  suggested  that  the  development  team  ensure  that  data  sources  are  complete  and 
contain  all  information  required  by  the  product.  Also,  user  assistance  messages  should  be  developed 
to  address  cases  in  which  the  data  source  is  incomplete  to  permit  the  users  to  proceed  with  their  tasks. 
Future  tests  need  to  be  conducted  on  systems  that  have  a  configuration  typical  of  those  used  by  the 
expected  “normal”  end-user  to  ensure  that  everything  works  as  intended. 
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INTERFACE  BUGS 


OVERVIEW 

This  section  contains  interface  bugs  found  during  study  for  this  report.  During  the  rapid  user 
testing  sessions  conducted  August  11,  2015,  users  saw  an  extra  line  in  the  “project  team  members” 
table,  in  which  no  information  was  populated.  Instead,  there  was  a  hyperlink  labeled  “Click  to  add 
role(s)”  highlighted  in  yellow,  as  shown  in  Figure  D-10  and  Figure  D-l  1. 


Project 

Lait  Nam# 

Flflt  Nam# 

Project  Team  Members 

Select  a  Project  vj 

AocijUfw  Team  Remove  export  List 

Wmtu  Si'lnlnl 

Co<J#  Email  Phone  Roi«»  Remove 

ClitV  to  j-JJ  roM*) 

[  ] 

[ - 1 

[ - ] 

«r  1  USERNAME@NAVY.MIL  [ 

Figure  D-10.  Interface  bug  example  one. 


Project  Team  Members 

laktl  4  PlufMl  j  w| 


Amricw  1CH1I 
UaiCsi 


Mrrmtc 


npen 


EjiujI 

USERNAME@NAVY.MIL 


•ben»  RcIm  fiMM 


Ui:»  »c  rr»3*f  -e-B  | 


Figure  D-1 1 .  Interface  bug  example  two. 


In  this  line  is  a  link  that  opens  an  interactive  window  for  assigning  roles.  Activating  the  link  leads 
to  an  error  message.  This  link  is  unexpected  and  confusing  for  users  because  it  causes  an  interaction 
without  any  possible  positive  resolution.  One  user  believed  the  bug  shown  in  Figure  D-10  and  Figure 
D-l  1  was  an  additional  method  of  adding  a  team  member.  The  UCD&E  team  concurs  this  method  of 
adding  a  team  member  would  be  a  beneficial  feature  if  it  worked.  However,  in  the  existing  portal 
software  it  does  not  work  and  leads  to  a  dead  end  in  the  portal  software.  Notice  the  placement  of  the 
extra  line  was  different  in  each  case,  as  well.  UCD&E  recommended  that  this  line  be  fixed  or 
removed  before  further  testing. 

Additionally,  in  Figure  D-10  and  D-l  1,  the  icons  above  the  “Add”,  “Remove”,  and  “Export”  links 
are  missing.  This  bug  only  manifests  in  certain  configurations  of  hardware  and  software.  Key  icons 
and  specific  navigation  and  environmental  cues  appear  to  be  missing,  depending  on  which  computer 
and  which  browser  is  used.  All  tests  were  performed  using  Internet  Explorer®  (IE)  10,  and  although 
the  UCD&E  team  tried  to  control  for  as  many  platform  environments  as  possible  (make  and  model, 
version  of  IE,  laptop  vs.  desktop),  various  key  visuals  were  not  visible  throughout  the  tests.  The 
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UCD&E  team  suggested  that  the  development  team  conduct  tests  on  typical  user  systems 
configuration  using  multiple  browsers  to  ensure  that  all  visual  and  interaction  elements  work  the 
same  across  browsers,  regardless  of  system,  browser,  or  other  variable  (like  network).  As  of  this 
writing,  the  UCD&E  team  has  not  discovered  the  reasons  for  the  missing  icons,  but  recommended 
that  the  development  team  follow  up  on  the  issue.  This  solution  should  be  a  high  priority  as 
additional  users  may  have  the  same  problems. 
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Figure  D-12  shows  the  protocol  script  used  during  rapid  user  testing  scenario  testing  conducted 
October  1,  2015. 


Thank  you  for  making  time  in  your  schedule  for  us! 

My  name  is _ .  I’m  a  Human  Factors  Engineer  in  the  UCD&E  group  and  we’re  helping 

design  a  suite  of  tools  to  help  support  our  project  managers. 

This  is _ s/he’ll  be  taking  notes  throughout  our  session. 

Today  we’d  like  to  show  you  an  incomplete  version  of  a  website  that  is  one  vision  of  what  a  new 
Center  process  tool  might  look  like.  These  are  early  designs  so  some  of  it  will  be  incomplete. 
There  will  be  elements  that  have  fairly  good  detail,  and  there  will  also  be  elements  that  will  be 
completely  missing. 

During  this  session,  we  will  be  taking  notes  (and  recording)  in  order  to  make  a  record  of  our 
session.  These  notes  and  recordings  are  only  for  our  reference.  They  are  simply  materials  we 
will  use  as  we  revise  our  design  and  update  User  Requirements.  None  of  the  information  you 
share  with  us  will  be  attributed  to  you  and  will  be  anonymized  in  any  supporting  documentation 
or  presentations  we  develop  as  a  result  of  this  session. 

While  we  look  at  these  pages  together,  I  want  to  remind  you  that  you're  helping  us  test  the  site 
design  -  we’re  not  testing  you.  You  are  giving  us  valuable  insights  that  will  allow  us  to  help 
users  like  you.  So  be  as  detailed  and  honest  as  you  can,  and  try  not  to  be  nervous.  This  is  not  a 
test  of  your  abilities,  but  a  test  of  the  page  design.  If  something’s  not  clear  to  you,  that’s  OUR 
fault. 

What  we  are  going  to  do  today  is  called  a  “task-based  analysis”.  What  we  will  do  is  give  you  a 
scenario  similar  to  something  that  might  actually  happen  in  real  life,  with  a  couple  extremely 
simple  tasks  to  perform.  We  would  like  you  to  complete  these  tasks  as  best  as  you  can  with  the 
interface  we  are  looking  at.  During  this  time,  we  will  be  observing  simply  as  a  means  of  gaining 
insight  into  the  usability  of  the  tool  as  currently  designed. 

The  most  important  part  about  your  participation  in  this  session  is  that  you  consider  each  task 
carefully  and  complete  it  as  best  you  can. 

Finally,  it’s  important  for  you  to  know  that  you  may  stop  this  session  at  any  point.  Similarly,  if  I 
ask  you  to  do  something  you  cannot  do,  for  any  reason,  simply  tell  me  and  I'll  move  on.  It’s 
important  that  you  feel  comfortable  throughout  this  process.  If  you  want  to  stop  at  any  time,  just 
let  me  know. 

Before  we  begin,  do  you  have  any  questions  or  concerns  about  this  process? 

Are  you  ready  to  begin? 

Figure  D-12.  Protocol  script. 
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Figure  D-13  shows  the  rapid  user  testing  survey  used  during  rapid  user  testing  scenario  testing 
conducted  October  1,  2015. 

_ Participant  Code  _ SSC-Pacific  Code 

Please  select  your  highest  degree  earned 

□  High  School 

□  Bachelors 

□  Post  Graduate 

□  Masters 

□  Doctorate 

□  Post  Doctorate 

Area  of  study: _ 

How  many  years  have  you  spent  in  government  service? 

_ years 

How  long  have  you  been  a  project  manager? 

□  Less  than  1  year 

□  1-5  years 

□  6-10  years 

□  11 -15  years 

□  16+ 

On  average,  how  many  projects  do  you  typically  manage  at  the  same  time?  Please 
include  any  projects  where  you  are  listed  as  the  project  manager  or  integrated  product 


team  lead 

□ 

1 

□ 

2-3 

□ 

4-5 

□ 

6-10 

□ 

11  + 

How  often  do  you  delegate  project  management  tasks  like  filling  out  the  WAT  or  PIT  to 
another  person? 

O  Always 
O  %  of  the  Time 
O  Half  of  the  Time 
O  !4  of  the  Time 
O  Never 

Figure  D-13.  Rapid  user  testing  survey. 
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CONCLUSION 


Virtually  everything  the  UCD&E  team  tested,  beyond  what  was  documented  above,  tested 
extremely  well.  Users  successfully  completed  all  tasks  given  them,  apart  from  those  exceptions. 
Overall,  the  three  rounds  of  user  testing  was  sufficient  to  capture  the  large  majority  of  user 
experience  issues  in  what  the  UCD&E  team  was  able  to  test.  The  pace  was  fast  enough  to  prevent  the 
development  team  from  doing  unnecessary  rework  in  many  cases,  but  moderate  enough  for  success 
with  a  smaller  team.  The  UCD&E  team  would  have  liked  to  do  more  testing  on  other  areas  of  the 
design,  but  due  to  schedule  and  personnel  constraints  in  the  development  team,  this  was  unrealistic. 

The  development  team  did  a  “wonderful”  job  coding  the  permissions  and  certificates  for  logging 
into  the  project  portal.  Both  types  of  CAC  certifications  (regular  and  email)  work  for  users  logging 
into  the  portal.  This  feature  gives  users  easy  access  into  to  the  portal  and  less  frustration,  delay,  and 
annoyance  due  to  failed  logins. 

The  “skipped”,  “complete”,  and  “in  progress”  labels  to  be  adopted  within  the  project  dashboard 
need  to  float  and/or  stick  as  users  scroll  down  through  the  rest  of  the  page.  This  recommendation  was 
explicitly  described  as  part  of  the  original  design,  and  remains  an  integral  part  of  efforts  to  improve 
user  experience  before  final  release  to  the  general  SSC  Pacific  user  population. 

Finally,  there  is  a  great  degree  of  learning  present  in  our  rapid  user  testing.  As  each  participant 
repeated  tasks,  the  time  and  effort  required  to  complete  the  task  was  reduced  with  each  repetition. 
Repeated  performances  resulted  in  easier  and  quicker  task  completion.  This  approach  is  a  key  finding 
of  rapid  user  testing,  and  a  major  success  as  well  users  can  quickly  become  “experts”  in  performing 
basic  tasks  after  two  or  three  instances  of  doing  the  same  thing  in  the  software.  This  result  suggests 
that  the  site,  as  designed,  will  lead  to  improved  experience  and  greater  performance  capabilities 
among  most  of  our  user  population. 


USER-CENTERED  DESIGN  AND  ENGINEERING  TEAM 
CONTACT  INFORMATION 

•  Deborah  Gill-Hesselgrave  deborah.d.gill@navy.mil 

•  Roni  Higgins  roni.higgins@navy.mil 

•  Sarah  Hunt  sarah.hunt@navy.mil 

•  Jake  Viraldo  jacob.viraldo@navy.mil 
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SCREEN  MAP  OF  PORTAL  ELEMENTS 


OVERVIEW 

The  User-Centered  Design  and  Engineering  (UCD&E)  team  worked  with  developers  to  improve 
the  portal  software.  This  appendix  contains  reference  material  distributed  electronically  to  developers 
that  was  used  as  a  guide  in  developing  the  portal  software.  Figure  numbers  were  used  to  provide 
targeted  feedback  for  specific  portal  software  regions,  icons,  screen  graphics  and  menus.  Numbers 
were  also  used  to  track  details  of  feature  implementation.  Number  tracking  details  are  in  Appendix  F. 

Portal  Software  Areas  Evaluated 

Following  is  a  breakdown  of  figures  in  this  section. 

Landing  Page 

The  Fanding  Page  is  shown  in  Figures  E-l  and  E-2. 

Projects 

My  Projects  Grid  View  is  shown  in  Figure  E-6.  Project  Cards  layout  specification  is  shown  in 
Figure  E-7.  Examples  of  two  Project  Cards  are  shown  in  Figures  E-10  and  E-l  1.  My  Project  (Fist 
View)  is  shown  in  Figure  E-8.  Figure  E-9  shows  the  Project  Status  Filter. 

Project  Dashboard 

The  Project  Dashboard  is  shown  in  Figure  E-l 9.  An  example  of  using  the  Project  Dashboard  to 
delegate  a  function  is  shown  in  Figure  E-20.  An  example  using  the  Project  Dashboard  to  join  or  leave 
a  project  is  shown  in  Figure  E-l 7. 

Project  Teams 

All  My  Project  Teams  is  shown  in  Figure  E-12.  Filters  for  Team  Tables  is  shown  in  Figure  E-13. 

Roles  Wizard 

The  roles  wizard  is  shown  in  Figures  E-l 4  to  E-l 5. 

Navigation 

Global  Navigation  is  shown  in  Figure  E-4.  Manage  Persistent  Navigation  is  shown  on  Figure  E-5. 

Portal  Flow  Diagrams 

Flow  diagrams  are  shown  on  Figures  E-l  1,  E-20  and  E-29. 
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Other  Portal  Figures 

The  Project  Dashboard  is  shown  in  Figures  E-19  to  E-21.  The  Workspace  Display  is  shown  in 
Figures  E-22  and  E-23.  Explore  Learn,  Practice  and  Browse  are  shown  in  Figures  E-27,  E-28  and 
E-30.  An  example  of  the  Help  screen  is  shown  in  Figure  E-16.  The  Footer  is  shown  in  Figure  E-3. 
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The  Landing  Page  is  shown  in  Figures  E-l  and  E-2. 


Landing  Page 


Close  up  screenshots  on 
following  slides. 


Figure  E-1 .  Landing  Page  (part  one). 
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Figure  E-2.  Landing  Page  Continued  (part  two). 
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One  of  four  footers  is  shown  in  Figure  E-3. 


Footer (1/4) 


Figure  E-3.  Footer  (one  of  four). 


An  example  of  Global  Navigation  is  shown  in  Figure  E-4. 


Global  Navigation 


Monoo*  or»d  EiptofO  i*om  up  one*  you  got 


Figure  E-4.  Global  Navigation. 
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An  example  Manage  Persistent  Navigation  is  shown  on  Figure  E-5. 


Manage  Persistent  Navigation 


User  Name:  First  Lost 
Welcome ,  First  Last! 


6 


Figure  E-5.  Manage  Persistent  Navigation. 


An  example  of  My  Projects  Grid  View  is  shown  in  Figure  E-6. 


My  Projects  Grid  View  (Default) 


Figure  E-6.  My  Projects  Grid  View  (Default). 
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Project  Cards  are  shown  in  Figures  E-7  to  E-12. 


Project  Cards 

Templote  Project  Cord 
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Figure  E-7.  Project  Cards  layout  specification. 


My  Projects  (List  View) 


My  Projects  JL39 

40 
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Figure  E-8.  My  Projects  (List  View). 
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Project  Status  Filter 


Current  Filter  (08/17/15)  Updated  Filter  42 


Stotus 

Wtert  Acceptorce  Incomplete  |v 

Active 

OoMd 

Wert  Acceplonce  Completed 
doped 

Figure  E-9.  Project  Status  Filter. 

An  example  of  the  Project  Card:  Joining  a  Project  (through  invitation  or  request)  is  shown  in 
Figures  E- 1 0  and  E- 1 1 . 
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Project  Card:  Joining  a 
Project  (Through  Request) 


If  Join  is  Accepted  by  PM 


Request  Sent-  PM  Decision  Pending 


Spocelab  Success  Green 


Project  Short  Nome 


Aft «r  uMr  « 
odO*d  th*  cord  a 


Figure  E-1 1 .  Project  Card:  Joining  a  Project  (Through  Request). 
The  All  My  Project  Teams  screen  is  shown  in  Figure  12. 


All  My  Project  Teams 


50 


All  My  Project  Teoms  52 
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Figure  E-1 2.  All  My  Project  Teams. 
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Filters  for  Team  Tables  is  shown  in  Figure  13. 


The  Roles  Wizard  is  shown  in  Figure  E-14,  three  different  design  options  are  shown  in  la,  lb,  and 
lc. 


Roles  Wizard 

As  per  discussions  during  Sprint  3  Demo  on  August  19th,  la  and  lb  will  be  combined 

la  “  lb  lc 


la  and  lb  need  tooltip  hovers  with  brief  descriptions  of  the 
roles  (from  PAIP). 


Figure  E-14.  Roles  Wizard. 
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The  Roles  Wizard  Tooltip  Hovers  is  shown  in  Figure  E-15. 


Roles  Wizard  Tooltip  Hovers 


{T/f  Assign  Project  Portol  &  PIT  y 
^  Roles 

Tm  not  sure’  gets  a  fnecxjy 
re mmder  they  can  do  this 
later 

Do  any  of  these  rotes  also  apply  tf Oonr  r*.  w-o,. 

O  r»"  not  sue  nght  now  K- 

O  PIT  Rote  PtocehotOsr  57 

O  PIT  Rote  PtocehotOar^w-- - 

O  PIT  Roto  PtoceftotOoj  «...  uT—— — 

O  (>rrRci»Pioc«*o!o4  " 

O  PIT  Rote  Placeholder 

These  rotes  w«  have  a  hover 
toot-tp  w*th  2-3  sentences 
provided  by  PWP  on  what  the 
roles  mean 

a  a 

Visually  showing  the  tooltip  hovers  on  lb. 


Figure  E-15.  Roles  Wizard  Tooltip  Hovers. 
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An  example  of  a  Help  screen  is  shown  in  Figure  E-16. 


Panels  for  the  different 
content  PAIP  will  be 
providing. 


Figure  E-16.  Help. 

Two  navigation  examples  are  shown  in  Figures  E-17  and  E-18. 
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Open  the  PIT  for  that  project  in  o 
NEW  TAB. 


Figure  E-17.  Project  Space  Persistent  Nav. 
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Left-Hand  Nav  Elements 
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Figure  E-18.  Left-Hand  Nav  Elements. 
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The  Project  Dashboard  is  shown  in  Figures  E-19  to  E-21. 


Figure  E-19.  Project  Dashboard. 

Project  Dashboard:  Team  Member  Requests  is  shown  in  Figure  E-20. 


Project  Dashboard:  Team  Member  Requests 

PM  and  PM  Delegate  Function  zzizzsr'"'  ®  ® 


Figure  E-20.  Project  Dashboard:  Team  Member  Requests  PM  and  PM  Delegate  Function. 
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The  PM  Dashboard  Notification  Join/Leave  Project  is  shown  in  Figures  E-21. 


PM  Dashboard  Notification  Join/Leave  Project 


Join 


Leave 


<firet  Lost>  is  requests^  to  jon  the 
<ProjectNome>  project 


79 


Accepted 

- ► 


<FrstLost>  is  no*  on  the  <ProjectNome> 
project  os  a  teom  member 


<First  Lost>  is  requesting  to  leove  the 
<ProjectNome>  project 


Accepted 

- ► 


You  hove  success! ufly  accepted  <FrstLast> 
leovsig  the  <ProjectNome>  project 


firveh 


OR 


OR 


Declined 

- ► 


You  hove  soccesfuty  deefcned  oddng 
<Frstt.ost>  to  the  <projectnome>  project 


Declined 

- ► 


You  hove  successful  dedrted  <FrstLast> 
leovng  the  <ProjectNome>  project 


Rwh 


If  the  users  chooses  to  update  the  team  member  role  from 
here,  it  pulls  up  the  same  roles  wizard  we  see  under  my 
teams. 
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Figure  E-21 .  PM  Dashboard  Notification  Join/Leave  Project. 
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Workspace  @  Phase  Level  is  shown  in  Figure  E-22  and  Workspace  Display  and  Activity  Level  is 
shown  in  Figure  E23. 


Workspace  @ 
Phase  Level 
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Figure  E-22.  Workspace  @  Phase  Level. 
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Workspace 
Display  @ 
Activity  Level 


Please  take  note: 


Activity  Tosks  (•) 


To»h  1 
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Figure  E-23.  Workspace  Display  @  Activity  Level. 
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The  Project  Team  (within  a  Project  Card),  is  shown  in  Figure  E-24. 


Project  Team  (within  Project 
Card) 


60 

Project  Nome 

Til 


85 


PieiKt  XXXXX't  T«m 


88 
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Figure  E-24.  Project  Team  (within  Project  Card). 
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The  Join  a  Project  -  Searching  by  Project  Name  is  shown  in  (Figure  E-25)  email  invitation  is 
shown  in  (Figure  E-26). 


Join  a  Project  -  Searching  by  Project  Name 


Figure  E-25.  Join  a  Project  -  Searching  by  Project  Name. 


Auto-generated  Email 
Template-Join  a 
Project  by  Invitation 


Subject  Line:  Invitation  to  Join  Project  <ProjectName> 

From:  SSC  Pacific  Project  Portal  (Automated  Message) 

To:  Username 

Content: 

<PMName>  has  invited  you  to  join  Project  <ProjectName>  in  the 
Project  Portal  as  a  <PortalRole>. 

To  accept  this  invitation,  please  follow  the  link  below. 

cHyperlinkto  user's  My  Projects  page> 


Subject:  Invitobon  to  Join  Project  <ProjectNome> 
FROM  SSC  Pacific  Project  Portol  (Automated  Message) 

To  Your  Name 

<PMNome>  has  invited  you  to  join  Project  <ProjectName>  in 
the  Project  Portol  as  o  <PortolRo»e> 

To  occept  this  invitation,  please  follow  the  Ink  below 

<Hyperlink  to  user's  My  Projects  poge> 


Respectfully 

SSC  Pocific's  Project  Portol  Teom 


Respectfully, 

SSC  Pacific's  Project  Portal  Team 


This  email  needs  to  be  generated  whenever  a  PM 
invites  ( adds j  someone  to  their  projects. 
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Figure  E-26.  Auto-generated  Email  Template-Join  a  Project  by  Invitation. 
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Explore  functions  are  shown  in  Figures  E-27,  E-28  and  E-30. 


Explore-Learn 


This  middle  jumbotron 
may  be  removed  as  per 
our  discussions  during 
the  Sprint  3  Demo, 
August  19th 


PAIP  content,  but  again  a 
long  scroll  layout  similar  to 
the  landing  page,  probably 
with  jumbotrons. 


I'd  expect  this  to  flip  between 
thumbnail  sets. 


PAIP  we  can  adjust  this  content  how  you'd  like,  the  goal 
is  to  provide  an  overview  of  where  the  ROIs  are  for  the 
PM  and  some  basic  info  about  what  a  process 
framework  is. 


10( 


Explore  Cent  2  r  'oject  Process  Best 


102  103  .IOI 


Motoofl  it  tor 


Figure  E-27.  Explore-Learn. 


Explore-Practice 


This  leads  into  the  sandbox. 
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Figure  E-28.  Explore-Practice. 
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Sandbox  Banner  &  Navigation 


The  practice  quiz  always  opens  the  sandbox  in  a 
new  tab,  maintaining  where  they  are  in 
the  interface  in  the  original  tab. 


Ytxx  demo  project  vnl  open  n  q  new 

tab 


A  tooltip  thot  they  ore  opening  o 
new  tab  should  appear  on  the 
mouse  hover  at  the  end  of  the  quiz 
(Exptore-Proc  t»ce) 


End  of  the  Quiz 


r 


114 


Take  me  to  my  Demo  Project! 


•poee  bock  to  the  porld  or  eaplore 


Figure  E-29. Sandbox  Banner  and  Navigation. 
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Figure  E-30.  Explore-Browse. 
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IMPLEMENTATION  STATUS  TABLE 


OVERVIEW 

This  section  contains  details  for  the  implementation  status  for  the  Project  Portal  User-Centered 
Design  &  Engineering  report.  Table  1  is  based  on  an  Excel®  spreadsheet  and  is  broken  down  into 
five  sections. 

Table  F-l  implementation  breakdown  is  as  follows: 

•  Numbers  in  the  Item  #  column  are  directly  related  to  the  feature  numbers  on  the  figures  in 
Appendix  E. 

•  The  first  four  table  rows  after  the  Item  #  column  track  implementation  status. 

•  The  Page  column  shows  which  portal  page  the  feature  appears. 

•  The  Element  column  shows  the  function  that  the  feature  does  on  the  page. 

•  Any  issues  specific  to  the  feature  are  shown  in  the  Issue  column. 

•  If  issues  were  found,  any  recommended  corrective  actions  are  in  the  Recommendation  row. 

•  The  Impact  column  tracks  impact  of  issues  in  levels:  low,  medium,  and  high. 

•  The  effect  on  the  portal  user  is  shown  in  the  Consequence  row. 

Table  F-2  contains  a  summary  for  implementation  status  as  of  August  25,  2015  for  all  numbered 
items  in  Table  F-l.  Figure  F-l  shows  a  pie  chart  comparing  implementation  status  results  by  percent. 
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CORRELATION  OF  APPENDIX  E  AND  APPENDIX  F 


Appendix  E  and  Appendix  F  are  directly  related  to  each  other  in  that  both  link  to  the  same 
numbers.  The  relationship  of  the  numbers  in  the  figures  in  Appendix  E  and  Table  F-l  are  shown  in 
Figure  F- 1 . 
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Figure  F-1 .  Appendix  E  and  F  number  correlations. 
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Table  F-1 .  Implementation  status  as  of  August  25,  201 5. 


Item  # 

Fully 

Implemented 

(100%) 

Mostly 
Implemented 
(50-90  %) 

Partially 

Implemented 

(49-1%) 

Not 

Implemented 

(0%) 

Page 

Element 

Issue 

Recommendation 

Impact 

Consequence 

1 

X 

PP 

Return  to  Home  Nav 

After  clicking,  dotted  box  around  the  link  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

2 

X 

PP 

Welcome  ID 

1  -  Name  is  all  capital  letters 

2  -  After  clicking,  dotted  box  around  the  link 
is  retained 

1  -  Name  should  be  in  sentence  case 

2  -  Make  sure  box  around  button  is 

cleared  after  click 

Low 

1  -  Inconsistent  with  the  rest  of  the  sentence  (poor 
mechanics);  does  not  match  "real-world”  heuristic; 
when  sets  of  names  are  in  all  capitals,  scanning  for 
finding  is  impeded,  thus  reducing  findability  of  a 
particular  name. 

2  -  Impacts  professionalism;  discourages  trust  in 
product 

3 

X 

PP 

Icon 

4 

X 

PP 

Heading 

5 

X 

PP 

Intro 

6 

X 

PP 

Manage  Anchor 

After  clicking,  box  around  the  anchor  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

Low 

Impacts  professionalism;  discourages  trust  in  product 

7 

X 

PP 

Explore  Anchor 

After  clicking,  box  around  the  anchor  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

Low 

Impacts  professionalism;  discourages  trust  in  product 

8 

X 

PP 

Manage 

This  control  and  functionality  is  missing  from 
the  developed  product 

Low-Medium 

As  the  portal  grows  to  include  future  functionality, 
these  jump  controls  support  users'  navigational  abilities 
and  serve  as  visual  anchors  on  the  page  and  to  help 
visually  dilineate  between  sections 

9 

X 

PP 

Manage  Icon 

10 

X 

PP 

Join  Button 

11 

X 

PP 

My  Projects  Button 

12 

X 

PP 

Create  Button 

13 

X 

PP 

Explore 

This  control  and  functionality  is  missing  from 
the  developed  product 

Low-Medium 

As  the  portal  grows  to  include  future  functionality, 
these  jump  controls  support  users'  navigational  abilities 
and  serve  as  visual  anchors  on  the  page  and  to  help 
visually  dilineate  between  sections 

14 

X 

PP 

Explore  Icon 

15 

X 

PP 

Learn  Button 

16 

X 

PP 

Practice  Button 

17 

X 

PP 

Browse  Button 

18 

X 

PP 

Footer  Links 

Footer  doesn’t  match  design 

rewrite 

High 

contact  information  and  links  need  to  be  present;  site 
map  provides  users  with  additional  navigation;  supports 
branding;  may  enhance  trust  in  product 

19 

X 

Global  NAV 

Manage  link 

After  clicking,  dotted  box  around  the  link  is 

retained 

Make  sure  box  around  button  is 

cleared  after  click 

Low 

Impacts  professionalism;  discourages  trust  in  product 

20 

X 

Global  NAV 

Explore  link 

After  clicking,  dotted  box  around  the  link  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

Low 

Impacts  professionalism;  discourages  trust  in  product 

21 

X 

Manage  NAV 

Title 

22 

X 

Manage  NAV 

Icon 

23 

X 

Manage  NAV 

User  Name 

Name  is  all  capital  letters 

Name  should  be  in  sentence  case 

Low 

Inconsistent  with  the  rest  of  the  sentence  (poor 
mechanics);  does  not  match  "real  world"  heuristic 

24 

X 

Manage  NAV 

My  Projects  link 

After  clicking,  dotted  box  around  the  link  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

Low 

Impacts  professionalism;  discourages  trust  in  product 

25 

X 

Manage  NAV 

My  Teams  link 

After  clicking,  dotted  box  around  the  link  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

Low 

Impacts  professionalism;  discourages  trust  in  product 

26 

X 

Manage  NAV 

Help  link 

After  clicking,  dotted  box  around  the  link  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

Low 

Impacts  professionalism;  discourages  trust  in  product 

27 

X 

Manage  NAV 

Create  a  Project 

28 

X 

My  Projects 

Workspace 

Project  Cards  are  displayed  in  no  discernible 
default  order 

Table  F-1 .  Implementation  status  as  of  August  25,  2015.  (continued). 


F-5 


Item  # 

Fully 

Implemented 

(100%) 

Mostly 
Implemented 
(50-90  %) 

Partially 

Implemented 

(49-1%) 

Not 

Implemented 

(0%) 

Page 

Element 

Issue 

Recommendation 

Impact 

Consequence 

29 

X 

My  Projects 

Project  cards 

30 

X 

Project  Card 

Short  Name 

31 

X 

Project  Card 

Logo 

32 

X 

Project  Card 

Content 

1  -  Temporary  ID  number  is  visible  to  user 

2  -  Process/ Activity  language  is  unclear 

1  -  Remove  T1D#  from  display 

2  -  Change  "Activity"  to  "Last 

Activity  Completed" 

1  -  Low-Medium 

2  -  Medium 

1  -  Confusing  and/or  unimportant  to  user.  Unneeded 
information.  Should  be  transparent  to  user. 

2  -  Creates  uncertainty  in  understanding  temporal 
awareness,  Need  to  use  precise  language. 

33 

X 

Project  Card 

PM  Name 

1  -  "PM"  is  used  instead  of  "Project 

Manager" 

2  -  Names  displayed  in  all  capitals 

1  -  "Project  Manager"  needs  to  be 
spelled  out 

2  -  Sentence  case  capitilization 
should  be  used  for  names 

1  -  Low 

2  -  Low 

1  -  Acronym  can  be  ambiguous  to  users 

2  -  Inconsistent  (poor  mechanics);  does  not  match  "real 
world"  heuristic 

34 

X 

New  Project 

Short  Name 

35 

X 

New  Project 

Logo 

36 

X 

Button 

Create  a  Project 
(Workspace) 

37 

X 

Button 

Join  a  Project 

38 

X 

My  Projects 

List  View  Selector 

Icon  is  not  visible  in  some  configurations 
using  Internet  Explorer 

Troubleshoot  and  fix 

Low 

Loss  of  salience;  confusion;  inconsistency;  discourages 
trust  in  product 

39 

X 

My  Projects 

Grid  View  Selector 

Icon  is  not  visible  in  some  configurations 
using  Internet  Explorer 

Troubleshoot  and  fix 

Low 

Loss  of  salience;  confusion;  inconsistency;  discourages 
trust  in  product 

... 

Not 

In 

Delivered 

Design 

My  Projects 

Export  List 

Icon  is  not  visible  in  some  configurations 
using  Internet  Explorer 

Troubleshoot  and  fix 

Low 

Loss  of  salience;  confusion;  inconsistency;  discourages 
trust  in  product 

40 

X 

My  Projects 

Card/List  Sort 

Does  not  sort  by  project  characteristics  as 
designed 

Add  sort-by-charateristic 
functionality  as  recommended  in 
delivered  design 

Medium-High 

Large  data  sets  are  difficult  to  manage,  inadequate 
sorting  capability  denies  users  a  useful  tool  for 
managing  data 

41 

X 

My  Projects 

List  View  Table 

1  -  No  Create  affordance 

2  -  No  Join  affordance 

3  -  No  ability  to  sort  by  column 

1  -  List  view  needs  a  way  to  enable 
user  to  Create  a  project,  either 
through  a  button  or  line  at  the 
bottom  of  the  table 

2  -  List  view  needs  a  way  to  enable 
user  to  Join  a  project,  either  through 
a  button  or  line  at  the  bottom  of 

the  table 

3  -  Add  sort-by-charateristic 
functionality  as  recommended  in 
delivered  design 

1  -  Medium 

2  -  Medium 

3  -  Medium-High 

1  -  User  must  navigate  to  another  part  of  the  interface 
to  accomplish  task 

2  -  User  must  navigate  to  another  part  of  the  interface 
to  accomplish  task 

3  -  Large  data  sets  are  difficult  to  manage,  inadequate 
sorting  capability  denies  users  a  useful  tool  for 
managing  data 

42 

X 

My  Projects 

Project  Status  Filter 

Imprecise  language,  ambiguous  terminology 

Change  "Pending",  "Active",  and 
"Closed"  as  directed  in  delivered 
design 

Low- Medium 

Users  may  not  inherently  understand  what  these  terms 
mean.  Providing  more  descriptive  language  enhances 
clarity. 

43 

X 

Join  A  Project 

Respond  to  Alert 

Not  currently  implemented  -  users 
mandated,  not  invited  to  projects 

Create  as  recommended  in 
delivered  design 

Low-Medium 

44 

X 

Join  A  Project 

Project  invitation 
Menu 

Not  currently  implemented  -  users 
mandated,  not  invited  to  projects 

Create  as  recommended  in 
delivered  design 

Low- Medium 

45 

X 

Join  A  Project 

Joined  Alert 

Not  currently  implemented  -  users 
mandated,  not  invited  to  projects 

Create  as  recommended  in 
delivered  design 

Low-Medium 
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Table  F-1 .  Implementation  status  as  of  August  25,  201 5.  (continued). 


Item  # 

Fully 

Implemented 

(100%) 

Mostly 
Implemented 
(50-90  %) 

Partially 

Implemented 

(49-1%) 

Not 

Implemented 

(0%) 

Page 

Element 

Issue 

Recommendation 

Impact 

Consequence 

46 

X 

Join  A  Project 

Declined  to  Join  Alert 

Not  currently  implemented  -  users 
mandated,  not  invited  to  projects 

Create  as  recommended  in 
delivered  design 

Low-Medium 

47 

X 

Join  A  Project 

Join  Pending  Alert 

Not  currently  implemented  -  Join  a  Project 
wizard  still  under  development 

Create  as  recommended  in 
delivered  design 

Low-Medium 

48 

X 

Join  A  Project 

Join  Accepted  Alert 

Not  currently  implemented  -  Join  a  Project 
wizard  still  under  development 

Create  as  recommended  in 
delivered  design 

Low-Medium 

49 

X 

Join  A  Project 

Join  Rejected  Alert 

Not  currently  implemented  -  Join  a  Project 
wizard  still  under  development 

Create  as  recommended  in 
delivered  design 

Low-Medium 

50 

X 

My  Teams 

Workspace 

Projects  are  displayed  in  no  discernible 
default  order  (though  Project  Team 

Members  are  displayed  alphabetically  within 
a  project) 

Default  display  Team  members 
alphabetically  within  projects 
displayed  alphabetically 

Medium 

Large  data  sets  are  difficult  to  manage,  random  default 
display  of  content  is  confusing. 

51 

X 

My  Teams 

Add  TM  Button 

52 

Not 

In 

Delivered 

Design 

My  Teams 

Remove  TM  Button 

53 

X 

My  Teams 

Download  Roster 

Button 

54 

X 

My  Teams 

My  Teams  Roster 

Table 

55 

X 

My  Teams 

My  Teams  Roster 
Filter/Sort 

Does  not  filter  or  sort  by  personnel 
attributes  characteristics  as  designed 

Add  filter/sort  functionality  as 
recommended  in  delivered  design 

Medium-High 

Large  data  sets  are  difficult  to  manage,  inadequate 
sorting  capability  denies  users  a  useful  tool  for 
managing  data 

56 

X 

Add  Team 

Member 

Wizard 

Search  results  are  not  ordered 

Order  search  results  alphabetically 

Low-Medium 

Difficulties  and  oversight  finding  the  correct  person  in  a 
high-population  search 

57 

X 

Add  Team 

Member 

Hover  Descriptions 

Not  implemented 

Implement 

Low-Medium 

Guides  users  in  correctly  choosing  desired  options 

58 

Help  Menu 

Content 

Not  implemented 

Implement 

Low-Medium 

Help  is  useful  for  many  users;  impact  can  be  lessened 
with  appropriate  contact  info  and  support 

59 

X 

Project 

Dashboard  NAV 

Project  Logo  (NAV) 

60 

X 

Project 

Dashboard  NAV 

Project  Name 

Project  name  is  all  capital  letters 

Change  to  Sentence  Case 

Low 

Affects  consistency;  breaks  ettiquette  of  professional 
written  language;  unnecessary 

61 

X 

Project 

Dashboard  NAV 

PMT# 

Temporary  ID  still  shows  for  pending 
projects. 

Remove  TID#  from  display 

Low-Medium 

Confusing  and/or  unimportant  to  user.  Unneeded 
information.  Should  be  transparent  to  user. 

62 

X 

Project 

Dashboard  NAV 

PIT  Button 

PIT  Button  moves  to  middle  of  page  when 
project  dashboard  is  being  displayed  on  a 

narrow  screen 

Make  sure  the  PIT  button  stays 
stable  to  the  right  of  the  screen  in 
all  display  modes 

Low-Medium 

Moving  buttons  can  be  confusing,  but  are  mostly  just 
unprofessional 

63 

X 

Left-Hand  NAV 

Project  Dashboard 

Nav 

64 

X 

Left-Hand  NAV 

Process  Nav 

1  -  Icons  (checks,  xs,  and  carrots)  don’t 
appear  in  all  computer  environments 

2  -  Suggested  color  scheme  not  used 

1  -  make  sure  icons  appear  in  all 
computer  environments 

2  -  Adopt  color  scheme 
recommended  in  design  packet 

1  -  Medium 

2  -  Medium-High 

1  -  Affects  usability  of  navigational  elements  by 
reducing  visibility  (change  blindness);  lower  situation 

awareness 

2  -  Affects  usability  of  navigational  elements  by 
reducing  visibility  (change  blindness);  lower  situation 

awareness 

65 

X 

Left-Hand  NAV 

Project  Team  Nav 

Link  goes  nowhere 

link  to  project-specific  roster  of 
team  members 

Medium-High 

Causes  users  to  be  unable  to  use  the  link  that  met 
users'  expectations  and  mental  model  the  most  during 
user  testing 

66 

Left-Hand  NAV 

Toolkit  Nav 

Unsure,  need  to  reevaluate  when  a  stable 
design  is  available 

67 

Left-Hand  NAV 

References  Nav 

Unsure,  need  to  reevaluate  when  a  stable 
design  is  available 

68 

X 

Project 

Dashboard  WS 

Project  Information 
Jumbotron 

Project  name  is  all  capital  letters 

Change  to  Sentence  Case 

Low 

Affects  consistency;  breaks  ettiquette  of  professional 
written  language;  unnecessary 
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Table  F-1 .  Implementation  status  as  of  August  25,  201 5.  (continued). 


Item  # 

Fully 

Implemented 

(100%) 

Mostly 
Implemented 
(50-90  %) 

Partially 

Implemented 

(49-1%) 

Not 

Implemented 

(0%) 

Page 

Element 

Issue 

Recommendation 

Impact 

Consequence 

69 

X 

Project 

Dashboard  WS 

Project  Information 

Data 

70 

X 

Project 

Dashboard  WS 

Update/Change 

Buttons 

They  don't  work 

Fix 

Medium 

Causes  user  confusion  and  distrust,  affects 
professionalism  of  design 

71 

X 

Project 

Dashboard  WS 

Team  Requests 

Notification  Bar 

72 

- 

Project 

Dashboard  WS 

Accordian-style 

Collapsing 

Unsure,  need  to  reevaluate  when  a  stable 
design  is  available 

73 

X 

Project 

Dashboard  WS 

Project  Logo 
(Workspace) 

74 

Project 

Dashboard  WS 

Process  Framework 

Overview 

1  -  PFO  colors  do  not  match  final  design 

2  -  Descriptions  only  appear  on  click,  not 
hover 

3  -  User  must  click  again  to  hide  description 

1  -  Change  colors  to  that  in  design 

2  -  Descriptions  appear  on  hover 
only,  non-clickable 

3  -  Descriptions  appear  on  hover 
only,  non-clickable 

1  -  Low 

2  -  Low-Medium 

3  -  Low-Medium 

1  -  Multicolored  interface  draws  users'  attention 

unnecessarily 

2  -  Doesn't  match  real  world  best  practices  for  websites 

3  -  Doesn't  match  real  world  best  practices  for  websites 

75 

X 

Project 

Dashboard  WS 

Process  Change 
Notifications 

76 

X 

Project 

Dashboard  WS 

Activity  Feed  Filter 

1  -  "Everyone"  filter  is  last 

2  -  Names  are  in  capital  letters 

1  -  "Everyone"  filter  should  be  first 

2  -  Change  to  Sentence  Case 

1  -  Low-Medium 

2  -  Low 

1  -  Eighth  numerous  team  members,  the  "Everyone" 
filter  may  be  hidden  or  difficult  to  find;  breaks  users' 

real-world  mental  model 

2  -  Affects  consistency;  breaks  ettiquette  of 
professional  written  language;  unnecessary 

77 

X 

Project 

Dashboard  WS 

Activity  Feed  Content 

78 

X 

Project 

Dashboard  WS 

PITT  Content 

79 

X 

Project 

Dashboard  WS 

Team  Request 

Wizard 

1  -  Actions  taken  on  notifications  don't  work 

2  -  Notifications  don't  disappear  after  taking 
action 

1  -  Allow  users  to  respond  to 
notifications  sucessfully 

2  -  Make  notifications  disappear 
after  action  is  taken 

1  -  Medium-High 

2  -  Medium 

1  -  Causes  user  confusion  and  distrust,  affects 
professionalism  of  design 

2  -  Notification  window  becomes  crowded  over  time, 
reduces  in  usability 

80 

Project  Process 

WS 

Content 

Unsure,  need  to  reevaluate  when  a  stable 
design  is  available 

81 

X 

Project  Process 
WS 

Print  Button 

82 

Project  Process 

WS 

Accordian-style 

Collapsing 

Unsure,  need  to  reevaluate  when  a  stable 
design  is  available 

83 

X 

Project  Process 
Activity 

Status  Selector 

1  -  Words  include  "waived"  and 
"incomplete" 

2  -  Bar  doesn't  "stick"  to  top  of  page  as  user 
scrolls 

1  -  Change  to  "Skipped"  and  "In 
Progress" 

2  -  Make  the  Status  selector  bar 
"stick"  during  scroll 

1  -  Low-Medium 

2  -  Low-Medium 

1  -  Confusion  and  loss  of  meaning  occurs  with  imprecise 
language 

2  -  Loss  of  situational  awareness;  confusion;  difficulty  in 
completing  the  task 

84 

X 

Project  Process 
Activity 

Content 

85 

X 

Project  Team 

Roster 

Workspace 

86 

X 

Project  Team 
Roster 

Add  TM  Button 

87 

X 

Project  Team 

Roster 

Remove  TM  Button 

88 

X 

Project  Team 
Roster 

Download  Roster 

Button 

89 

X 

Project  Team 

Roster 

Project  Team  Roster 
Table 

90 

X 

Project  Team 

Roster 

Project  Team  Roster 
Filter/Sort 
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Table  F-1 .  Implementation  status  as  of  August  25,  201 5.  (continued). 


Item  # 

Fully 

Implemented 

(100%) 

Mostly 
Implemented 
(50-90  %) 

Partially 

Implemented 

(49-1%) 

Not 

Implemented 

10*) 

Page 

Element 

Issue 

Recommendation 

Impact 

Consequence 

91 

X 

Create  A  Project 

Wizard 

1  -  Project  Manager's  name  is  not  listed 
during  creation 

2  -  Wording  doesn't  match  design 

3  -  Buttons  don't  match  design 

1  -  List  PM  name  during  creation,  as 
recommended  in  design  packet 

2  -  Adopt  wording  in  design 

3  -  Make  sure  buttons  describe 
literal  action  beiong  taken 

111 

1  -  Potential  to  cause  confusion;  doesn't  provide 
enough  user  assurance  during  the  task 

2  -  Doesn't  provide  enough  user  assurance  during  the 
task 

3  -  Potential  to  cause  confusion;  doesn't  properly 
manage  user  expectations  during  the  task 

92 

X 

Create  A  Project 

Icon 

No  icon  present 

Use  recommended  icon 

93 

X 

Create  A  Project 

Input 

94 

X 

Create  A  Project 

Confirmation 

No  confirmation  window  present 

Add  confirmation  window,  as 
recommended  in  design  packet 

Medium 

Potential  to  cause  confusion;  doesn't  provide  enough 
user  assurance  upon  task  completion 

95 

X 

Req  to  Join  A 
Project 

Wizard 

The  ability  to  search  by  PMT  number  is  not 
available 

Allow  users  to  search  by  PMT 
number 

Low 

May  increase  difficulty  to  user  to  find  desired  project 

96 

X 

Req  to  Join  A 
Project 

Icon 

97 

Req  to  Join  A 
Project 

Input 

Unsure,  need  to  reevaluate  when  a  stable 
design  is  available 

98 

X 

Req  to  Join  A 
Project 

Confirmation 

99 

X 

Email 

Notifications 

Content 

Doesn't  match  language  recommended  in 
design  packet 

Adonpt  language  as  reccomended  in 
design  packet 

Low 

The  user  is  not  fully  informed  about  what  they  are 
supposed  to  do  and  what  will  happen  next 

100 

X 

Explore  NAV 

Title 

101 

X 

Explore  NAV 

Icon 

102 

X 

Explore  NAV 

Learn  link 

After  clicking,  dotted  box  around  the  link  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

103 

X 

Explore  NAV 

Practice  link 

After  clicking,  dotted  box  around  the  link  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

104 

X 

Explore  NAV 

Browse  link 

After  clicking,  dotted  box  around  the  link  is 
retained 

Make  sure  box  around  button  is 

cleared  after  click 

105 

X 

Learn 

Contentl 

106 

X 

Learn 

Content2 

107 

X 

Practice 

Title 

108 

X 

Practice 

Icon 

109 

X 

Practice 

Type  Selector 

110 

X 

Practice 

Type  Slide 

Placeholder  icons  not  updated 

Replace  with  approved  icons 

Low 

Inconsistent  with  overall  design 

111 

X 

Practice 

Characterization 

Placeholder  icons  not  updated 

Replace  with  approved  icons 

Low 

Inconsistent  with  overall  design 

112 

X 

Practice 

Characterization 

Selector 

113 

X 

Practice 

Link  to  Demo 

114 

X 

Demo 

Banner 

115 

X 

Demo 

User  Name 

116 

X 

Browse 

Contentl 

117 

X 

Browse 

Content2 

118 

• 

Browse 

References 

Unsure,  need  to  reevaluate  when  a  stable 
design  is  available 

Totals 

40 

31 

5 

32 
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Table  F-2.  Summary  table  for  implementation  status  as  of  August  25,  2015. 


Implementation  Status 

Percent  completed  (%) 

Fully  Implemented 

33.90 

Mostly  Implemented 

26.27 

Partially  Implemented 

4.24 

Not  Implemented 

27.12 

Unsure  (TBD  later) 

7.63 

A  pie  chart  showing  a  comparative  implementation  status  is  provided  in  Figure  F-2. 


■  Fully  Implemented 

■  Mostly 
Implemented 

■  Partially 
Implemented 

■  Not  Implemented 

■  Unsure  (TBD  later) 


Figure  F-2.  Comparative  implimentation  status  chart. 
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